-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
420 lines (397 loc) · 21.4 KB
/
index.html
File metadata and controls
420 lines (397 loc) · 21.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="description"
content="View-consistent Object Removal in Radiance Fields.">
<meta name="keywords" content="Multiview Segmentation, Neural Radiance Fields, 3D Editing">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>View-consistent Object Removal in Radiance Fields</title>
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
rel="stylesheet">
<link rel="stylesheet" href="./static/css/bulma.min.css">
<link rel="stylesheet" href="./static/css/bulma-carousel.min.css">
<link rel="stylesheet" href="./static/css/bulma-slider.min.css">
<link rel="stylesheet" href="./static/css/fontawesome.all.min.css">
<link rel="stylesheet"
href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<link rel="stylesheet" href="./static/css/index.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap@4.3.1/dist/css/bootstrap.min.css"
integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script defer src="./static/js/fontawesome.all.min.js"></script>
<script src="./static/js/bulma-carousel.min.js"></script>
<script src="./static/js/bulma-slider.min.js"></script>
<script src="./static/js/index.js"></script>
</head>
<body>
<section class="hero">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-1 publication-title">View-consistent Object Removal in Radiance Fields</h1>
<div class="is-size-5 publication-authors">
<!-- Paper authors -->
<span class="author-block">
<a href="https://yiren-lu.com/" target="_blank">Yiren Lu</a>,</span>
<span class="author-block">
<a href="https://jma712.github.io/" target="_blank">Jing Ma</a>,</span>
<span class="author-block">
<a href="https://yin-yu.github.io/" target="_blank">Yu Yin</a>
</span>
</div>
<div class="is-size-5 publication-authors">
<span class="author-block">Case Western Reserve University<br>ACM MM 2024</span>
<!-- <span class="eql-cntrb"><small><br><sup>*</sup>Indicates Equal Contribution</small></span> -->
</div>
<div class="column has-text-centered">
<div class="publication-links">
<!-- Arxiv PDF link -->
<span class="link-block">
<a href="https://arxiv.org/pdf/2408.02100v1.pdf" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-file-pdf"></i>
</span>
<span>Paper</span>
</a>
</span>
<!-- Supplementary PDF link -->
<!-- <span class="link-block">
<a href="static/pdfs/supplementary_material.pdf" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-file-pdf"></i>
</span>
<span>Supplementary</span>
</a>
</span> -->
<!-- ArXiv abstract Link -->
<span class="link-block">
<a href="https://arxiv.org/abs/2408.02100v1" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="ai ai-arxiv"></i>
</span>
<span>arXiv</span>
</a>
</span>
<!-- Github link -->
<!-- <span class="link-block">
<a href="https://github.com/YOUR REPO HERE" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-github"></i>
</span>
<span>Code (Coming soon)</span>
</a>
</span> -->
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<!-- Teaser image-->
<section class="hero teaser">
<div class="container is-max-desktop">
<div class="hero-body">
<div class="columns is-centered has-text-centered">
<h2 class="title is-3">Abstract</h2>
</div>
<div class="item">
<!-- Your image here -->
<img src="static/images/teaser.png" alt="MY ALT TEXT"/>
</div>
<div class="content has-text-justified">
<p>
Radiance Fields (RFs) have emerged as a crucial technology for 3D scene representation, enabling the synthesis of novel views with remarkable realism. However, as RFs become more widely used, the need for effective editing techniques that maintain coherence across different perspectives becomes evident. Current methods primarily depend on per-frame 2D image inpainting, which often fails to maintain consistency across views, thus compromising the realism of edited RF scenes.
In this work, we introduce a novel RF editing pipeline that significantly enhances consistency by requiring the inpainting of only a single reference image. This image is then projected across multiple views using a depth-based approach, effectively reducing the inconsistencies observed with per-frame inpainting. However, projections typically assume photometric consistency across views, which is often impractical in real-world settings. To accommodate realistic variations in lighting and viewpoint, our pipeline adjusts the appearance of the projected views by generating multiple directional variants of the inpainted image, thereby adapting to different photometric conditions.
Additionally, we present an effective and robust multi-view object segmentation approach as a valuable byproduct of our pipeline. Extensive experiments demonstrate that our method significantly surpasses existing frameworks in maintaining content consistency across views and enhancing visual quality.
</p>
</div>
</div>
</section>
<!-- End teaser image -->
<!-- pipeline overview -->
<section class="section hero is-light">
<div class="container is-max-desktop">
<div class="hero-body">
<div class="columns is-centered has-text-centered">
<h2 class="title is-3">Inpainting Pipeline</h2>
</div>
<div class="item">
<img src="static/images/framework.png" alt="MY ALT TEXT"/>
</div>
<div class="content has-text-justified">
<p>
An overview of our proposed pipeline. We initiate our methodology by selecting a reference camera pose from the training dataset; this camera pose is identified as having the minimal average distance to all other poses on the SE(3) manifold. The processing of the chosen reference view involves three key steps: masking, inpainting, and depth estimation, yielding three outputs: the mask, the inpainted image, and the depth map, respectively. These outputs are then used for multi-view projection, yielding a set of inpainted images from multiple views. Finally, an inpainted Radiance Field will be trained using these inpainted images.
</p>
</div>
</div>
</section>
<!-- End pipeline overview -->
<section class="hero is-small">
<div class="hero-body">
<div class="container has-text-centered">
<h2 class="title">Scene Object Removal Results</h2>
<div id="results-carousel" class="carousel results-carousel">
<div class="item rounded-0 border-0 item-1">
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/ori_1_spiral_050000_rgb.mp4" type="video/mp4">
</video>
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/1_removed_gs.mp4" type="video/mp4">
</video>
</div>
<div class="item rounded-0 border-0 item-2">
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/ori_2_spiral_050000_rgb.mp4" type="video/mp4">
</video>
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/2_removed_gs.mp4" type="video/mp4">
</video>
</div>
<div class="item rounded-0 border-0 item-3">
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/ori_3_spiral_050000_rgb.mp4" type="video/mp4">
</video>
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/3_removed_gs.mp4" type="video/mp4">
</video>
</div>
<div class="item rounded-0 border-0 item-4">
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/ori_4_spiral_050000_rgb.mp4" type="video/mp4">
</video>
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/4_removed_gs.mp4" type="video/mp4">
</video>
</div>
<div class="item rounded-0 border-0 item-7">
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/ori_7_spiral_050000_rgb.mp4" type="video/mp4">
</video>
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/7_removed_gs.mp4" type="video/mp4">
</video>
</div>
<div class="item rounded-0 border-0 item-9">
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/ori_9_spiral_050000_rgb.mp4" type="video/mp4">
</video>
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/9_removed_gs.mp4" type="video/mp4">
</video>
</div>
<div class="item rounded-0 border-0 item-10">
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/ori_10_spiral_050000_rgb.mp4" type="video/mp4">
</video>
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/10_removed_gs.mp4" type="video/mp4">
</video>
</div>
<div class="item rounded-0 border-0 item-12">
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/12_ori.mp4" type="video/mp4">
</video>
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/12_removed_gs.mp4" type="video/mp4">
</video>
</div>
<div class="item rounded-0 border-0 item-book">
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/ori_book_spiral_050000_rgb.mp4" type="video/mp4">
</video>
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/book_removed_gs.mp4" type="video/mp4">
</video>
</div>
<div class="item rounded-0 border-0 item-trash">
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/ori_trash_spiral_050000_rgb.mp4" type="video/mp4">
</video>
<video poster="" autoplay muted loop preload="none" class="w-auto" style="height:22vh;">
<source src="static/videos/trash_removed_gs.mp4" type="video/mp4">
</video>
</div>
</div>
</div>
</div>
</section>
<br/>
<section class="hero teaser">
<div class="container is-max-desktop has-text-centered shadow-sm">
<br/>
<h2 class="title">Comparison with Previous Methods</h2>
<div class="row">
<div class="col-3">
<video poster="" autoplay muted loop height="100%" class="rounded" preload="none">
<source src="static/videos/ori_3_spiral_050000_rgb.mp4"
type="video/mp4">
</video>
</div>
<div class="col-3">
<video poster="" autoplay muted loop height="100%" class="rounded" preload="none">
<source src="static/videos/3_lpips_True_prepare_False_010000_rgb.mp4"
type="video/mp4">
</video>
</div>
<div class="col-3">
<video poster="" autoplay muted loop height="100%" class="rounded" preload="none">
<source src="static/videos/3_nerfiller.mp4"
type="video/mp4">
</video>
</div>
<div class="col-3">
<video poster="" autoplay muted loop height="100%" class="rounded" preload="none">
<source src="static/videos/3_removed_gs.mp4"
type="video/mp4">
</video>
</div>
</div>
<div class="row">
<div class="col-3">
<video poster="" autoplay muted loop height="100%" class="rounded" preload="none">
<source src="static/videos/ori_book_spiral_050000_rgb.mp4"
type="video/mp4">
</video>
<h5 class="subtitle">Original Scene</h5>
</div>
<div class="col-3">
<video poster="" autoplay muted loop height="100%" class="rounded" preload="none">
<source src="static/videos/book_lpips_True_prepare_False_010000_rgb.mp4"
type="video/mp4">
</video>
<h5 class="subtitle">SPIn-NeRF</h5>
</div>
<div class="col-3">
<video poster="" autoplay muted loop height="100%" class="rounded" preload="none">
<source src="static/videos/book_nerfiller.mp4"
type="video/mp4">
</video>
<h5 class="subtitle">NeRFiller</h5>
</div>
<div class="col-3">
<video poster="" autoplay muted loop height="100%" class="rounded" preload="none">
<source src="static/videos/book_removed_gs.mp4"
type="video/mp4">
</video>
<h5 class="subtitle">Ours</h5>
</div>
</div>
</div>
</section>
<br/>
<section class="hero is-light">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="column is-centered ">
<h2 class="title has-text-centered">Multi-view Consistency</h2>
<div class="row d-flex justify-content-around">
<div class="content has-text-justified">
<p class="content">Here we showcase how our pipeline ensures the multi-view consistency by evaluating the number of keypoint matchings across different rendered views. These keypoint matchings are given by SuperGlue, and we only consider the matchings within the inpainted region. As evident in the results, our pipeline has significantly more results within the inpainted region, which indicates a better multi-view consistency.</p>
</div>
<div class="row justify-content-around">
<div class="col-8">
<video poster="" autoplay muted loop height="100%" class="rounded" preload="none">
<source src="static/videos/book_spinnerf_matching.mp4"
type="video/mp4">
</video>
<h2 class="subtitle has-text-centered">SPIn-NeRF</h2>
</div>
</div>
<div class="row justify-content-around">
<div class="col-8">
<video poster="" autoplay muted loop height="100%" class="rounded" preload="none">
<source src="static/videos/book_nerfiller_matching.mp4"
type="video/mp4">
</video>
<h2 class="subtitle has-text-centered">NeRFiller</h2>
</div>
</div>
<div class="row justify-content-around">
<div class="col-8">
<video poster="" autoplay muted loop height="100%" class="rounded" preload="none">
<source src="static/videos/book_gs_matching.mp4"
type="video/mp4">
</video>
<h2 class="subtitle has-text-centered">Ours</h2>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<section class="hero teaser">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="column is-full">
<h2 class="title has-text-centered">Mask Consistency</h2>
<div class="row d-flex justify-content-around">
<div class="content has-text-justified">
<p class="content has-text-justified">Our multi-view segmentation approach can not only maintain mask consistency across different views, but can also take regions without semantic meanings into consideration (i.e. the shadow on the left side of the book).
</p>
</div>
<div class="col-5 p-1">
<video poster="" autoplay muted loop height="100%" class="rounded" preload="none">
<source src="static/videos/ori_book_spiral_050000_rgb.mp4"
type="video/mp4">
</video>
<h2 class="subtitle has-text-centered">Original Scene</h2>
</div>
<div class="col-5 p-1">
<video poster="" autoplay muted loop height="100%" class="rounded" preload="none">
<source src="static/videos/book_mask.mp4"
type="video/mp4">
</video>
<h2 class="subtitle has-text-centered">Masks</h2>
</div>
</div>
</div>
</div>
</div>
</section>
<section class="section" id="BibTeX">
<div class="container is-max-desktop content">
<h2 class="title">Citation</h2>
<p>If you find our work helpful, please consider cite us:</p>
<pre><code>@inproceedings{lu2024view,
title={View-consistent Object Removal in Radiance Fields},
author={Lu, Yiren and Ma, Jing and Yin, Yu},
booktitle={Proceedings of the 32nd ACM International Conference on Multimedia},
pages={3597--3606},
year={2024}
}
</code></pre>
</div>
</section>
<footer class="footer">
<div class="container">
<div class="columns is-centered">
<div class="column is-8">
<div class="content">
<p>Website template borrowed from <a href="https://spinnerf3d.github.io/">SPIn-NeRF</a> and <a href="https://github.com/ornerf/ornerf.github.io">OR-NeRF</a>.</p>
<p>
This website is licensed under a <a rel="license"
href="http://creativecommons.org/licenses/by-sa/4.0/">Creative
Commons Attribution-ShareAlike 4.0 International License</a>.
</p>
<!-- <p>
This means you are free to borrow the <a
href="https://github.com/ornerf/ornerf.github.io">source code</a> of this website,
we just ask that you link back to this page in the footer.
Please remember to remove the analytics code included in the header of the website which
you do not want on your website.
</p> -->
</div>
</div>
</div>
</div>
</footer>
</body>
</html>