“Transferring image-based edits for multi-channel compositing”
Conference:
Type(s):
Title:
- Transferring image-based edits for multi-channel compositing
Session/Category Title: HDR and Image Manipulation
Presenter(s)/Author(s):
Abstract:
A common way to generate high-quality product images is to start with a physically-based render of a 3D scene, apply image-based edits on individual render channels, and then composite the edited channels together (in some cases, on top of a background photograph). This workflow requires users to manually select the right render channels, prescribe channel-specific masks, and set appropriate edit parameters. Unfortunately, such edits cannot be easily reused for global variations of the original scene, such as a rigid-body transformation of the 3D objects or a modified viewpoint, which discourages iterative refinement of both global scene changes and image-based edits. We propose a method to automatically transfer such user edits across variations of object geometry, illumination, and viewpoint. This transfer problem is challenging since many edits may be visually plausible but non-physical, with a successful transfer dependent on an unknown set of scene attributes that may include both photometric and non-photometric features. To address this challenge, we present a transfer algorithm that extends the image analogies formulation to include an augmented set of photometric and non-photometric guidance channels and, more importantly, adaptively estimate weights for the various candidate channels in a way that matches the characteristics of each individual edit. We demonstrate our algorithm on a variety of complex edit-transfer scenarios for creating high-quality product images.
References:
1. 3DArtist. 2016. Compositing 3D render passes in Photoshop. https://www.youtube.com/watch?v=wrsvFKNgxgU. (2016).
2. Xiaobo An, Xin Tong, Jonathan D. Denning, and Fabio Pellacini. 2011. AppWarp: Retargeting Measured Materials by Appearance-space Warping. ACM SIGGRAPH Asia 30, 6 (Dec. 2011), 10.
3. Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing. ACM SIGGRAPH 28, 3 (July 2009), 11.
4. Connelly Barnes and Fang-Lue Zhang. 2017. A survey of the state-of-the-art in patch-based synthesis. Computational Visual Media 3, 1 (2017), 17. Cross Ref
5. Connelly Barnes, Fang-Lue Zhang, Liming Lou, Xian Wu, and Shi-Min Hu. 2015. PatchTable: Efficient Patch Queries for Large Datasets and Applications. ACM SIGGRAPH 34, 4 (Aug. 2015), 10.
6. Gilbert Louis Bernstein and Wilmot Li. 2015. Lillicon: Using Transient Widgets to Create Scale Variations of Icons. ACM SIGGRAPH 34, 4 (July 2015), 11.
7. Floraine Berthouzoz, Wilmot Li, Mira Dontcheva, and Maneesh Agrawala. 2011. A Framework for Content-adaptive Photo Manipulation Macros: Application to Face, Landscape, and Global Manipulations. ACM TOG 30, 5 (Oct. 2011), 14.
8. Ralph Allan Bradley and Milton E. Terry. 1952. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika 39, (3) (1952).
9. CGalter. 2015. Multipass Compositing in Photoshop – Vray Render Elements. https://www.youtube.com/watch?v=q0WpIzi8sc4. (2015).
10. Ming-Ming Cheng, Fang-Lue Zhang, Niloy J. Mitra, Xiaolei Huang, and Shi-Min Hu. 2010. RepFinder: Finding Approximately Repeated Scene Elements for Image Editing. ACM SIGGRAPH 29, 4, Article 83 (2010), 8 pages.
11. Ming-Ming Cheng, Shuai Zheng, Wen-Yan Lin, Vibhav Vineet, Paul Sturgess, Nigel Crook, Niloy J. Mitra, and Philip Torr. 2014. ImageSpirit: Verbal Guided Image Parsing. ACM TOG 34, 1, Article 3 (Dec. 2014), 11 pages.
12. Soheil Darabi, Eli Shechtman, Connelly Barnes, Dan B Goldman, and Pradeep Sen. 2012. Image Melding: Combining Inconsistent Images using Patch-based Synthesis. ACM SIGGRAPH 31, 4 (July 2012), 10.
13. Olga Diamanti, Connelly Barnes, Sylvain Paris, Eli Shechtman, and Olga Sorkine-Hornung. 2015. Synthesis of Complex Image Appearance from Limited Exemplars. ACM TOG 34, 2 (March 2015), 14.
14. FastCompany. 2014. 75% Of Ikea’s Catalog is Computer Generated Imagery. https://www.fastcodesign.com/3034975/75-of-ikeas-catalog-is-computer-generated-imagery. (2014).
15. Jakub Fišer, Ondřej Jamriška, Michal Lukáč, Eli Shechtman, Paul Asente, Jingwan Lu, and Daniel Sýkora. 2016. StyLit: Illumination-Guided Example-Based Stylization of 3D Renderings. ACM SIGGRAPH 35, 4 (July 2016), 11.
16. L. A. Gatys, A. S. Ecker, and M. Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. (June 2016), 9 pages.
17. Floraine Grabler, Maneesh Agrawala, Wilmot Li, Mira Dontcheva, and Takeo Igarashi. 2009. Generating Photo Manipulation Tutorials by Demonstration. ACM SIGGRAPH 28, 3 (July 2009), 9.
18. Paul Guerrero, Gilbert Bernstein, Wilmot Li, and Niloy J. Mitra. 2016. PATEX: Exploring Pattern Variations. ACM SIGGRAPH 35, 4 (July 2016), 13.
19. Paul Guerrero, Stefan Jeschke, Michael Wimmer, and Peter Wonka. 2014. Edit Propagation Using Geometric Relationship Functions. ACM SIGGRAPH 33, 2 (July 2014), 15.
20. Yoav HaCohen, Eli Shechtman, Dan B Goldman, and Dani Lischinski. 2011. Non-Rigid Dense Correspondence with Applications for Image Enhancement. ACM SIGGRAPH 30, 4 (July 2011), 9.
21. Yoav HaCohen, Eli Shechtman, Dan B Goldman, and Dani Lischinski. 2013. Optimizing Color Consistency in Photo Collections. ACM SIGGRAPH 32, 4 (July 2013), 9.
22. S.W. Hasinoff, M. Jozwiak, F. Durand, and W.T. Freeman. 2010. Search-and-replace editing for personal photo collections. In IEEE International Conference on Computational Photography. 8.
23. Paul S. Heckbert. 1990. Adaptive Radiosity Textures for Bidirectional Ray Tracing. ACM SIGGRAPH 24, 4 (Sept. 1990), 10.
24. James W. Hennessey and Niloy J. Mitra. 2015. An Image Degradation Model for Depth-augmented Image Editing. SGP 34, 5 (Aug. 2015), 9.
25. Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image Analogies. (Aug. 2001), 14 pages.
26. Shi-Min Hu, Kun Xu, Li-Qian Ma, Bin Liu, Bi-Ye Jiang, and Jue Wang. 2013a. Inverse Image Editing: Recovering a Semantic Editing History from a Before-and-after Image Pair. ACM TOG 32, 6 (Nov. 2013), 11.
27. Shi-Min Hu, Fang-Lue Zhang, Miao Wang, Ralph R. Martin, and Jue Wang. 2013b. PatchNet: A Patch-based Image Representation for Interactive Library-driven Image Editing. ACM TOG 32, 6 (Nov. 2013), 12.
28. L. Itti and C. Koch. 2001. Computational Modelling of Visual Attention. Nature Reviews Neuroscience 2, 3 (2001), 9. Cross Ref
29. Wenzel Jakob. 2010. Mitsuba renderer. (2010). http://www.mitsuba-renderer.org.
30. Anat Levin, Dani Lischinski, and Yair Weiss. 2004. Colorization Using Optimization. ACM SIGGRAPH 23, 3 (Aug. 2004), 6.
31. Ce Liu, Jenny Yuen, and Antonio Torralba. 2011. SIFT Flow: Dense Correspondence Across Scenes and Its Applications. IEEE PAMI 33, 5 (May 2011), 17.
32. Stephen Robert Marschner. 1998. Inverse rendering for computer graphics. Technical Report.
33. Morgan McGuire. 2011. Computer Graphics Archive. (August 2011). http://graphics.cs.williams.edu/data.
34. Fabio Pellacini, Frank Battaglia, Keith Morley, and Adam Finkelstein. 2007. Lighting with Paint. ACM TOG 26, 2 (June 2007), 10.
35. Ravi Ramamoorthi and Pat Hanrahan. 2001. A Signal-processing Framework for Inverse Rendering (ACM SIGGRAPH). 12.
36. Erik Reinhard, Michael Ashikhmin, Bruce Gooch, and Peter Shirley. 2001. Color Transfer between Images. IEEE Comput. Graph. Appl. 21, 5 (Sept. 2001), 8.
37. Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. 2004. “GrabCut”: Interactive Foreground Extraction Using Iterated Graph Cuts. ACM TOG 23, 3 (Aug. 2004), 6.
38. Thorsten-Walther Schmidt, Fabio Pellacini, Derek Nowrouzezahrai, Wojciech Jarosz, and Carsten Dachsbacher. 2016. State of the Art in Artistic Editing of Appearance, Lighting and Material. CGF 35, 1 (2016), 17.
39. TheFoundry. 2017. Nuke. https://www.thefoundry.co.uk/products/nuke/. (2017). https://www.thefoundry.co.uk/products/nuke/
40. Yonatan Wexler, Eli Shechtman, and Michal Irani. 2007. Space-Time Completion of Video. IEEE PAMI 29, 3 (March 2007).
41. Jun Xing, Hsiang-Ting Chen, and Li-Yi Wei. 2014. Autocomplete Painting Repetitions. ACM SIGGRAPH Asia 33, 6 (Nov. 2014), 11.
42. Kaan Yücer, Alec Jacobson, Alexander Hornung, and Olga Sorkine. 2012. Transfusive Image Manipulation. ACM SIGGRAPH Asia 31, 6 (Nov. 2012), 11.
43. Kaan Yücer, Alexander Sorkine-Hornung, and Olga Sorkine-Hornung. 2013. Transfusive Weights for Content-Aware Image Manipulation. CGF Eurographics.
44. F. L. Zhang, J. Wang, E. Shechtman, Z. Y. Zhou, J. X. Shi, and S. M. Hu. 2016. PlenoPatch: Patch-based Plenoptic Image Manipulation. IEEE TVCG 23, 5 (2016), 14.
45. Youyi Zheng, Xiang Chen, Ming-Ming Cheng, Kun Zhou, Shi-Min Hu, and Niloy J. Mitra. 2012. Interactive Images: Cuboid Proxies for Smart Image Manipulation. ACM SIGGRAPH 31, 4 (2012), 11.
