Being able to refocus a photograph after it has been captured is becoming a key trait of new smartphones. But a masters graduate of
MIT's Media Lab thinks he can do it better -- with a camera filter that costs less than ten pence.
Kshitij Marwah's technology is called Tesseract and stems from a technology he and three colleagues developed at the Media Lab around three years ago for SLR cameras. It was a filter that was created by inkjet printing onto a transparent film that could be inserted into a camera between the light sensor and the lens. Called Focii, it allowed the camera to record depth information into an image, which was then manipulable by software created for desktop computers by Marwah's team -- points of focus could be altered, or special effects could be applied to specific parts of an image such as a person in the foreground. "When you put a printed transparency between the lens and the sensor it attenuates the angles of incoming light rays with different weights," Marwah explains. The software can then "compute back" and "recover" individual light rays. "Once you have this angular information it helps you recover parallax [the apparent perspective of a scene from the camera] and hence the depth information about the scene."
Now he has developed the same technology in miniature for smartphone cameras. It works without having multiple camera lenses or the user having to manually add a filter themselves.
"I don't think anybody has had this sort of technology working on a mobile phone [before us]," Marwah, who has founded Tesseract Imaging in India to market the system, tells Wired.co.uk. "When we put the ten-cent (six pence) filter in the camera, it gives you 16 or more different perspectives of the same scene. When you have multiple perspectives of the same scene you can do something called correspondence matching and know what the depth is of each object, which is way better than using two cameras or two images. In this case it has so many different perspectives it knows what is in the front and what is in the back."
Marwah's comment about "two cameras" could be seen as a shot across the bow for numerous smartphone makers approaching post-capture refocussing and special effects in their devices. It is strongly rumoured HTC will release a flagship smartphone with refocussing technology stemming from the use of stereo camera lenses, and Samsung included a similar tool in its Galaxy S5 using one camera but multiple levels of focus at the point of image capture. Ironically, one Tesseract prototype was developed on an HTC Evo 3D -- an Android smartphone from 2011 that featured two lenses on the rear.
However, it was the Lytro camera that initially got Marwah excited originally. "Back when I was in MIT about two to three years back we started working on this technology and in 2011 Lytro had just launched," he says. "A couple of issues I saw with Lytro: one was loss of resolution; secondly it was pretty expensive.
"The advantage that we will always have as a company is the price point is extremely low. All you require is a filter printed out, which costs about ten cents."
There are drawbacks, however -- namely processing power. The images produced with Focii in SLR cameras required a lot of hardware oomph to manipulate. "You are recovering the whole resolution back, which we felt that in time, using cloud computing and GPUs, could be easily solved," Marwah says. This is only made more difficult on mobile phones, however, which inherently have to balance performance with battery life. "But we hope that the multicore processors will keep on improving so we can keep the
resolution."
Additionally, the nature of placing a filter over a sensor leaves artefacts. "Technically a photograph captured with the mask does have the mask imprinted on it," Marwah explained. "Though the mask pattern can be removed [in software] to get a clear 2D photograph."
Now it is the goal of Tesseract Imaging, which employs only three other people alongside Marwah, to get manufacturers including its filter directly within their camera modules. Those conversations with OEMs are about to begin, Wired.co.uk understands.
Marwah is already looking at the next steps: video ("If you could do this on video, I think it could give a lot of control that has been unseen"), and healthcare. "If you put this filter in a camera within an endoscope you can get 16 different perspectives of the same scene with depth and refocussing," he says. "It can add a lot of value to a surgeon or a doctor."
This article was originally published by WIRED UK
