“The Frankencamera: an experimental platform for computational photography” by Adams, Talvala, Park, Jacobs, Ajdin, et al. …

Although there has been much interest in computational photography within the research and photography communities, progress has been hampered by the lack of a portable, programmable camera with sufficient image quality and computing power. To address this problem, we have designed and implemented an open architecture and API for such cameras: the Frankencamera. It consists of a base hardware specification, a software stack based on Linux, and an API for C++. Our architecture permits control and synchronization of the sensor and image processing pipeline at the microsecond time scale, as well as the ability to incorporate and synchronize external hardware like lenses and flashes. This paper specifies our architecture and API, and it describes two reference implementations we have built. Using these implementations we demonstrate six computational photography applications: HDR viewfinding and capture, low-light viewfinding and capture, automated acquisition of extended dynamic range panoramas, foveal imaging, IMU-based hand shake detection, and rephotography. Our goal is to standardize the architecture and distribute Frankencameras to researchers and students, as a step towards creating a community of photographer-programmers who develop algorithms, applications, and hardware for computational cameras.

1. Adams, A., Gelfand, N., and Pulli, K. 2008. Viewfinder alignment. Computer Graphics Forum (Proc. Eurographics) 27, 2, 597–606.Google ScholarCross Ref
2. Adobe, Inc., 2010. The Digital Negative Format. http://www.adobe.com/products/dng/.Google Scholar
3. 2010. The Ångström Linux Distribution. http://www.angstrom-distribution.org/.Google Scholar
4. Bae, S., Agarwala, A., and Durand, F. 2010. Computational re-photography. ACM Transactions on Graphics (To appear). Google ScholarDigital Library
5. Bhat, P., Zitnick, C. L., Snavely, N., Agarwala, A., Agrawala, M., Cohen, M., Curless, B., and Kang, S. B. 2007. Using photographs to enhance videos of a static scene. In Proc. Eurographics Symposium on Rendering.Google Scholar
6. Bramberger, M., Doblander, A., Maier, A., Rinner, B., and Schwabach, H. 2006. Distributed embedded smart cameras for surveillance applications. Computer 39, 2, 68–75. Google ScholarDigital Library
7. 2010. The CHDK Project. http://chdk.wikia.com.Google Scholar
8. Chen, J., Paris, S., and Durand, F. 2007. Real-time edge-aware image processing with the bilateral grid. ACM Transactions on Graphics 26, 3 (July), 103:1–103:9. Google ScholarDigital Library
9. Debevec, P. E., and Malik, J. 1997. Recovering high dynamic range radiance maps from photographs. In Proceedings of ACM SIGGRAPH 1997, ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 369–378. Google ScholarDigital Library
10. Durand, F., 2009. private communication.Google Scholar
11. Eisemann, E., and Durand, F. 2004. Flash photography enhancement via intrinsic relighting. ACM Transactions on Graphics 23, 3 (Aug.), 673–678. Google ScholarDigital Library
12. Filippov, A. 2003. Reconfigurable high resolution network camera. In Field-Programmable Custom Computing Machines, 2003. FCCM 2003. 11th Annual IEEE Symposium on, 276–277. Google ScholarDigital Library
13. Greenberg, S., and Fitchett, C. 2001. Phidgets: easy development of physical interfaces through physical widgets. In UIST ’01: Proc. of the 14th annual ACM symposium on user interface software and technology, ACM, New York, NY, USA, 209–218. Google ScholarDigital Library
14. Hengstler, S., Prashanth, D., Fong, S., and Aghajan, H. 2007. Mesheye: a hybrid-resolution smart camera mote for applications in distributed intelligent surveillance. In IPSN ’07: Proceedings of the 6th international conference on Information processing in sensor networks, 360–369. Google ScholarDigital Library
15. Joshi, N., Kang, S. B., Zitnick, C. L., and Szeliski, R. 2010. Image deblurring using inertial measurement sensors. ACM Transactions on Graphics (Proc. SIGGRAPH) 29, 3 (Aug.). Google ScholarDigital Library
16. Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2003. High dynamic range video. In SIGGRAPH ’03: ACM SIGGRAPH 2003 Papers, ACM, New York, NY, USA, 319–325. Google ScholarDigital Library
17. Kleihorst, R., Schueler, B., Danilin, A., and Heijligers, M. 2006. Smart camera mote with high performance vision system. In ACM SenSys 2006 Workshop on Distributed Smart Cameras (DSC 2006).Google Scholar
18. 2010. The Maemo Linux Distribution. http://maemo.org/.Google Scholar
19. 2010. The Magic Lantern project. http://magiclantern.wikia.com/wiki/.Google Scholar
20. Mann, S., and Picard, R. W. 1995. On being ‘undigital’ with digital cameras: Extending dynamic range by combining differently exposed pictures. In Proceedings of IS&T, 442–448.Google Scholar
21. Mertens, T., Kautz, J., and Reeth, F. V. 2007. Exposure fusion. Proceedings of Pacific Graphics. Google ScholarDigital Library
22. Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., and Toyama, K. 2004. Digital photography with flash and no-flash image pairs. In SIGGRAPH ’04: ACM SIGGRAPH 2004 Papers, ACM, New York, NY, USA, 664–672. Google ScholarDigital Library
23. Rahimi, M., Baer, R., Iroezi, O., Garcia, J. C., Warrior, J., Estrin, D., and Srivastava, M. 2005. Cyclops: in situ image sensing and interpretation in wireless sensor networks. In SenSys ’05: Proceedings of the 3rd international conference on Embedded networked sensor systems, 192–204. Google ScholarDigital Library
24. Raskar, R., and Tumblin, J. 2010. Computational Photography: Mastering New Techniques for Lenses, Lighting, and Sensors. In Press. A K Peters, Natick, MA, USA. Google ScholarDigital Library
25. Raskar, R., Agrawal, A., and Tumblin, J. 2006. Coded exposure photography: motion deblurring using fluttered shutter. In SIGGRAPH ’06: ACM SIGGRAPH 2006 Papers, ACM, New York, NY, USA, 795–804. Google ScholarDigital Library
26. Reinhard, E., Ward, G., Pattanaik, S., and Debevec, P. 2006. High Dynamic Range Imaging – Acquisition, Display and Image-based Lighting. Morgan Kaufman Publishers, 500 San-some Street, Suite 400, San Francisco, CA 94111. Google ScholarDigital Library
27. Rowe, A., Goode, A., Goel, D., and Nourbakhsh, I. 2007. CMUcam3: An open programmable embedded vision sensor. Tech. Rep. RI-TR-07-13, Carnegie Mellon Robotics Institute, May.Google Scholar
28. Tico, M., and Pulli, K. 2009. Image enhancement method via blur and noisy image fusion. In ICIP ’09: IEEE International Conference on Image Processing, 1521–1525. Google ScholarDigital Library
29. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. In SIGGRAPH ’05: ACM SIGGRAPH 2005 Papers, ACM, New York, NY, USA, 765–776. Google ScholarDigital Library
30. Wolf, W., Ozer, B., and Lv, T. 2002. Smart cameras as embedded systems. Computer 35, 48–53. Google ScholarDigital Library
31. Yodaiken, V. 1999. The RTLinux Manifesto. In Proceedings of The 5th Linux Expo.Google Scholar