“Image-based street-side city modeling”
Conference:
Type(s):
Title:
- Image-based street-side city modeling
Session/Category Title: Urban modeling
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We propose an automatic approach to generate street-side 3D photo-realistic models from images captured along the streets at ground level. We first develop a multi-view semantic segmentation method that recognizes and segments each image at pixel level into semantically meaningful areas, each labeled with a specific object class, such as building, sky, ground, vegetation and car. A partition scheme is then introduced to separate buildings into independent blocks using the major line structures of the scene. Finally, for each block, we propose an inverse patch-based orthographic composition and structure analysis method for façade modeling that efficiently regularizes the noisy and missing reconstructed 3D data. Our system has the distinct advantage of producing visually compelling results by imposing strong priors of building regularity. We demonstrate the fully automatic system on a typical city example to validate our methodology.
References:
1. Barinova, O., Konushin, V., Yakubenko, A., Lim, H., and Konushin, A. 2008. Fast automatic single-view 3-d reconstruction of urban scenes. In Proceedings of the European Conference on Computer Vision, 100–113. Google ScholarDigital Library
2. Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 11, 1222–1239. Google ScholarDigital Library
3. Canny, J. F. 1986. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 8, 679–714. Google ScholarDigital Library
4. Cornelis, N., Leibe, B., Cornelis, K., and Gool, L. V. 2008. 3D urban scene modeling integrating recognition and reconstruction. International Journal of Computer Vision 78, 2, 121–141. Google ScholarDigital Library
5. Coughlan, J., and Yuille, A. 1999. Manhattan world: compass direction from a single image by bayesian inference. In Proceeding of IEEE International Conference in Computer Vision, vol. 2, 941–947. Google ScholarDigital Library
6. Curless, B., and Levoy, M. 1996. A volumetric method for building complex models from range images. In Proceedings of SIGGRAPH 96, ACM Press / ACM SIGGRAPH, H. Rushmeier, Ed., Computer Graphics Proceedings, Annual Conference Series, ACM, 303–312. Google ScholarDigital Library
7. de Berg, M., Cheong, O., van Kreveld, M., and Overmars, M. 2008. Computational Geometry: Algorithms and Applications, 3rd ed. Springer, Berlin. Google ScholarDigital Library
8. Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach. In Proceedings of SIGGRAPH 96, ACM Press / ACM SIGGRAPH, H. Rushmeier, Ed., Computer Graphics Proceedings, Annual Conference Series, ACM, 11–20. Google ScholarDigital Library
9. Dick, A., Torr, P., and Cipolla, R. 2004. Modelling and interpretation of architecture from several images. International Journal of Computer Vision 60, 2, 111–134. Google ScholarDigital Library
10. Felzenszwalb, P., and Huttenlocher, D. 2004. Efficient graph-based image segmentation. International Journal of Computer Vision 59, 2, 167–181. Google ScholarDigital Library
11. Frueh, C., and Zakhor, A. 2003. Automated reconstruction of building facades for virtual walk-thrus. In SIGGRAPH ’03: ACM SIGGRAPH 2003 Sketches&Applications, ACM, New York, NY, USA, 1–1. Google ScholarDigital Library
12. Früh, C., and Zakhor, A. 2003. Constructing 3d city models by merging ground-based and airborne views. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition, 562–569.Google Scholar
13. Geman, S., and Geman, D. 1984. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence 6, 6, 721–741.Google ScholarDigital Library
14. Hartley, R. I., and Zisserman, A. 2004. Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press. Google ScholarDigital Library
15. Hoiem, D., Efros, A. A., and Hebert, M. 2005. Automatic photo pop-up. ACM Transactions on Graphics 24, 3 (Aug.), 577–584. Google ScholarDigital Library
16. Lhuillier, M., and Quan, L. 2005. A quasi-dense approach to surface reconstruction from uncalibrated images. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 418–433. Google ScholarDigital Library
17. Müller, P., Zeng, G., Wonka, P., and Gool, L. V. 2007. Image-based procedural modeling of façades. ACM Transactions on Graphics (Aug.), 85:1–85:10. Google ScholarDigital Library
18. Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In Proceedings of SIGGRAPH 2001, ACM Press / ACM SIGGRAPH, E. Fiume, Ed., Computer Graphics Proceedings, Annual Conference Series, ACM, 433–442. Google ScholarDigital Library
19. Oliva, A., and Torralba, A. 2006. Building the gist of a scene: the role of global image features in recognition. Progress in Brain Research 155, Part 2, 23–36.Google ScholarCross Ref
20. Pearl, J. 1982. Reverend bayes on inference engines: a distributed hierarchical approach. In Proceedings of AAAI National Conference on AI, 133–136.Google Scholar
21. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Transactions on Graphics 22, 3 (Aug.), 313–318. Google ScholarDigital Library
22. Pollefeys, M., Nistér, D., Frahm, J., Akbarzadeh, A., Mordohai, P., Clipp, B., Engels, C., Gallup, D., Kim, S., Merrell, P., Salmi, C., Sinha, S., Talton, B., Wang, L., Yang, Q., Stewnius, H., Yang, R., Welch, G., and Towles, H. 2008. Detailed real-time urban 3D reconstruction from video. International Journal of Computer Vision 78, 2, 143–167. Google ScholarDigital Library
23. Saxena, A., Sun, M., and Ng, A. Y. 2009. Make3d: learning 3d scene structure from a single still image. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 5, 824–840. Google ScholarDigital Library
24. Schindler, G., Krishnamurthy, P., and Dellaert, F. 2006. Line-based structure from motion for urban environments. In 3D Data Processing, Visualization, and Transmission, Third International Symposium on, 846–853. Google ScholarDigital Library
25. Shotton, J., Winn, J., Rother, C., and Criminisi, A. 2009. TextonBoost for image understanding: multi-class object recognition and segmentation by jointly modeling texture, layout, and context. International Journal of Computer Vision 81, 1, 2–23. Google ScholarDigital Library
26. Sinha, S. N., Steedly, D., Szeliski, R., Agrawala, M., and Pollefeys, M. 2008. Interactive 3D architectural modeling from unordered photo collections. ACM Transactions on Graphics 27, 5 (Dec.), 159:1–159:10. Google ScholarDigital Library
27. Stamos, I., and Allen, P. K. 2002. Geometry and texture recovery of scenes of large scale. Computer Vision and Image Understanding 88, 2, 94–118. Google ScholarDigital Library
28. Torralba, A., Murphy, K., and Freeman, W. 2007. Sharing visual features for multiclass and multiview object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 5, 854–869. Google ScholarDigital Library
29. van den Hengel, A., Dick, A., Thormählen, T., Ward, B., and Torr, P. H. S. 2007. VideoTrace: rapid interactive scene modelling from video. ACM Transactions on Graphics 26, 3 (Aug.), 86:1–86:5. Google ScholarDigital Library
30. Werner, T., and Zisserman, A. 2002. Model selection for automated architectural reconstruction from multiple views. In Proceedings of the British Machine Vision Conference, 53–62.Google Scholar
31. Winn, J., Criminisi, A., and Minka, T. 2005. Object categorization by learned universal visual dictionary. In Proceedings of IEEE International Conference in Computer Vision, vol. 2, 1800–1807. Google ScholarDigital Library
32. Xiao, J., Fang, T., Tan, P., Zhao, P., Ofek, E., and Quan, L. 2008. Image-based façade modeling. ACM Transactions on Graphics 27, 5 (Dec.), 161:1–161:10. Google ScholarDigital Library
33. Zebedin, L., Klaus, A., Gruber-Geymayer, B., and Karner, K. 2006. Towards 3D map generation from digital aerial images. ISPRS Journal of Photogrammetry and Remote Sensing 60, 6 (Sep.), 413–427.Google ScholarCross Ref
