Studying animal behavior used to mean traveling into the wild and making detailed notes about gorillas. Now, biologist-coders are figuring out how to use computer vision techniques to convert the myriad motions of creatures large and small into crunchable data.
Researchers are figuring out how track the movements of insects such as Drosophila, the fruit fly, in order to answer the question: How do we define behavior?
"A fundamental problem that we haven't done that much work on in biology is quantifying behavior," said Kristin Branson, a fellow at Howard Hughes Medical Institute's Janelia Farm Research Campus. "We have a much better handle on the very low-level things, molecular, genetic, and neural than we do at the global, large-scale level of behavior."
We know what behavior is: It's what animals do. But quantifying it isn't easy, even for tiny creatures with equally tiny brains. Big data came to branches of science like particle physics many years ago, but some realms of biology have remained resistant to the computational techniques that mark so many other disciplines. The data for a lot of behavioral biology remains simple human observations – or results from ingenious Rube Goldbergian experimental apparatus. Either way, it's hard to do what Branson calls high-throughput behavioral experiments.
So, while researchers mapped the fruit fly's genome in 2000 and know its genetics better than almost any other creature, the relationship between its genes, its brain, and its behavior is still hard to understand.
At Janelia, Branson's lab head, Gerry Rubin, is mapping out the circuits in the fruit fly brain. His team has created thousands of transgenic flies that allow them to test the individual circuits. But while we know what we've done, it's hard to tell what it makes the the flies do.
Let's say some genetic change is made to the fruit flies and they chase one another around 20 percent more often than an unaltered specimen. If you're the fly, that's an important change, but how could a human researcher ever detect that 20 percent? It's not like counting how many times a monkey mother nurses.
"How do we say in a quantitative way how the behavior has changed?" Branson said. "You wouldn't notice that if you were just watching."
To solve that problem, Branson and collaborators in Michael Dickinson's lab at Caltech, where she was a postdoc, built the Caltech Multiple Fly Tracker. It's a piece of software that converts infrared video of up to 50 flies inside a special arena into movement data. The flies become small triangles in space and their behavior is plotted and recorded.
Another Dickinson lab postdoc, Andrew Straw, has even designed a 10-camera system he calls Flydra to track free moving, flying insects.
Some of what they've found is odd and unexpected. After recording male and female flies at Caltech, they mined the data for interesting differences between them.
"And if you looked at how often the fly turned, you could tell the gender of the fly with better than 90 percent accuracy," Branson said.
It's unclear why such a behavioral difference exists, but it does, and likely always has, hidden within the masses of data that our eyes receive when we watch a bunch of flies moving around.
All sorts of other behaviors emerge from the data, if you just watch for long enough.
"Fruit flies may not be as interesting as gorillas on the surface to humans. They just seem like little gnat sized things," said Serge Belongie, a computer vision specialist at the University of California San Diego, who was Branson's Ph.D. advisor. "But you run this tracker long enough and there is some pretty interesting courtship competitiveness behavior. It's basically reality TV for fruit flies with some interesting stuff happening."
"We're finding subtle differences between individual flies now," Branson agreed. "If you're being not very technical about things, you can say that these flies have different personalities. In biology we try not to do it, but it's a fun way to think of it."
While computer vision is more familiar to people as the technology behind Optical Character Recognition or social media applications, it may work better with animal tracking than it does in some other settings. That's because researchers can design experiments that make acquiring clean data easier.
By designing the algorithms and the image acquisition apparatus together, it makes the most difficult computer vision problems disappear.
"If you think about people-tracking, you can solve it at the 80 percent solved level because you don't have complete control of your environment," Branson said. "I want things to work at 99 or 100 percent. I feel like we can really solve the problem well enough that people will use these programs, and it will be a very clean solution."
While Branson's work qualifies as basic science, computer vision insect monitoring could have more immediate implications. Take beekeeping, which has been plagued by colony collapse disorder. Intel researcher Lily Mummert, a backyard apiarist, built a tracking tool that could identify bees coming and going from her own hive. Counting the number of bees coming into and out of it, and perhaps some other data, could yield important insights about the life and times of a beehive, she said.
Ideally, all the equipment could be miniaturized and stuck into a little unit that would beam data up.
"I'd like to see a little unit, a camera, full-on board processor, and a little wireless transmitter so you could just mule off the count," Mummert said. "That thing could be a really versatile platform for all kinds of environmental monitoring. You could apply it to bees, you could apply it to anything."
All kinds of insect- and animal-monitoring experts got together for a workshop in late 2008, and they plan to do it again this year in Istanbul during the International Conference on Pattern Recognition.
With video cameras and computational ability getting cheaper and better, quantifying animal behavior will undoubtedly improve. It's possible that before too long, there will be a new encyclopedia of knowledge on the biology block: the behaviorome.
WiSci 2.0: Alexis Madrigal's Twitter, Tumblr, and green tech history research site; Wired Science on Twitter and Facebook.**
