Watching people in conversation, you can usually tell when they lose interest or get confused or excited. But computers have a hard time telling a smile from a smirk, confusion from anger.
After examining thousands of faces from around the world, a Waltham start-up has figured out how to interpret these subtle shifts in expressions in real time. By reading tiny movements in the corners of the mouth, nose, and eyes and the shifting angles of the eyebrows, Affectiva Inc. can digitally decode facial messages — more effectively, it says, than others have done before.
The company's facial-coding product, Affdex, has been used by advertisers, including Coca-Cola and Unilever, to gauge consumers' reactions to commercials — one-fourteenth-of-a-second at a time.
Advertisement
On Friday, Affectiva released a beta version of Affdex for mobile devices, so companies will be able to get the same kind of feedback from cameras on consumer's smartphones and tablets — with their permission, of course.
Affectiva hopes the new platform will help broaden its customer base to include companies in the mobile gaming, online education, and health-tracking businesses, virtually any field where an app developer wants feedback.
"We understand that emotion is important in a whole bunch of applications," said Rana el Kaliouby, who helped develop the basis for the Affdex technology while she was a research scientist at the Media Lab at the Massachusetts Institute of Technology.
Although most people do not have trouble interpreting facial expressions, computers struggle to distinguish between an individual's features and his or her expressions, Kaliouby said.
At MIT and in early work at Affectiva, she developed an algorithm for identifying expressions. With a genuine smile, the mouth is curved equally on both sides, while it is uneven in a smirk, for example. Disgust differs from confusion — though both have a furrowed brow — with a wrinkled nose.
Advertisement
By digitally distinguishing between expressions, Affectiva's customers — so far, they are mainly marketing firms — can analyze test subjects' reactions as they watch ads. Understanding when these subjects lose interest, get confused, or are truly engaged allows companies to make more effective advertisements and sell more products.
Marketers have historically gotten this kind of data by hand-coding viewers' responses, but digital analysis is cheaper, faster, and now viable on a large scale, Kaliouby said.
Advertising was a logical first step for using the technology, Kaliouby and others said, because split-second reactions to an ad can mean the difference between a new customer and lost opportunity.
"Facial coding will pick up quite fleeting responses that people aren't able to recall later on, or they might not see as that important," said Sarah Walker, research and development director for the consumer neuroscience practice at Millward Brown, a global market research firm that has used Affdex to gauge consumers' responses to some 2,000 different ads in the past two years.
Sometimes people will report liking an ad, but when the firm asks them to watch it again, a slight frown, picked up only by digital facial coding, reveals their true feelings, Walker said. "Very fleeting responses like that can be incredibly predictive" of an ad's overall success.
It is probably fair to say that Kaliouby has the only technology company in Boston where some of the formative work is performed by female school-bus monitors in Cairo.
Advertisement
As part of her efforts to make the facial coding as accurate as possible, Kaliouby set up shop in her native Egypt, and then hired some two dozen women who rode the buses for the private school her children attended. The women used to sit bored in the school basement while the children were in class.
Now they work for her part time, scanning facial expressions to determine each fleeting emotion. Every expression is coded several times to ensure accuracy. So far, the women have individually coded 100,000 expressions, captured as people watch media. The women's coding is then used to train the software to accurately recognize a vast array of expressions from around the world.
As a young company, Affectiva recently underwent a painful transition. The four-year-old company started with twin technologies, both begun in the lab of MIT professor Rosalind Picard: the facial coding, and medical sensors that detect excitation, used for example to identify when nonverbal people with autism are getting overstimulated.
In the spring, Affectiva elected to focus solely on the facial coding business, and Picard said it was time for the two technologies to be split apart.
"A small company can't build a medical research business and a market research business" at the same time, said Picard, who is trying to start another company with the sensor technology.
But the decision has created problems for the researchers using the sensors.
"You create a technology out of necessity, you do good science with it, you make it widely available, and when the company decides to cease manufacturing, it stalls the science," said Matthew Goodwin, an assistant professor at Northeastern University and founding member of Affectiva's scientific advisory board.
Advertisement
He has hoarded 30 sensors to use in his ongoing studies of children with severe developmental disabilities, including autism. "We're now all dependent on the technology and there is not a good alternative in the marketplace, which was the very reason we commercialized it in the first place."
Affdex, meanwhile, says it has built up the world's largest database of coded expressions, and its analytics have been shown to be effective in 53 countries, according to Kaliouby and Walker of Millward Brown.
"The fact that we can deploy at scale is really the key thing," Walker said. "Our clients like the fact that they're able to use it worldwide and have consistent messages on all their advertising."