You Can't Always Believe Your Eyes: Where AI Falls Flat
In short
A facial expression does not reliably show what a person feels or thinks. A 2019 review of the research found that the same expression can mean different things across people and situations, and video adds its own distortions. A side glance that looked like doubt turned out to be an intern looking for his mouse.
You cannot reliably tell what someone feels or thinks from their face, whether you are a person or a piece of software. People do sometimes scowl when angry and smile when happy, but the same expression shows up for many other reasons, and the link changes from culture to culture, situation to situation and person to person. That was the finding of a large review of the research published in 2019 (Barrett et al., Psychological Science in the Public Interest). Video makes it harder still.
A story from Okaya's CEO.
I learned this the embarrassing way, watching a recording of our intern, Elvin, at the end of an exam.
What went wrong with the intern's video?
Near the end of the recording, Elvin's eyes slid to the side. I read the look on his face as doubt. I decided, right there, that the exam had gone badly. Thankfully I did not act on that conclusion.
Then I heard a click, and that click showed me how wrong I was.
Elvin was looking for his mouse. The side glance was him finding the cursor on his screen so he could stop the recording. He was not hiding anything about the exam. He aced it.
It is a small story, and the mistake was an ordinary one. I had one channel (a face on a screen), a strong expectation, and no context. My brain filled in the rest.
Can you tell if someone is lying from their eyes?
No. The popular rules about eye direction do not hold up when researchers test them. Websites still repeat claims such as "looking up to the right means a lie," which come from a practice called Neuro-Linguistic Programming.
A team led by Richard Wiseman tested exactly that claim in three studies, including footage of real liars and truth-tellers at high-profile press conferences. The eye movements did not match the pattern, and people taught the rule were no better at spotting lies than people who were not (Wiseman et al., PLOS ONE, 2012).
The habit of judging people by their faces is old. Physiognomy, the practice of judging character from appearance, goes back centuries. The tools changed. The temptation stayed the same: we all rush to judgment from limited information, me included.
Why are facial expressions a weak signal for emotion?
Facial movements carry real information, but one expression rarely maps to one emotion. Barrett and colleagues found that how people show anger, disgust, fear, happiness, sadness and surprise varies a great deal across cultures, situations and even across people in the same situation (Barrett et al., 2019).
They also found that a single configuration of the face, a scowl for example, often communicates something other than an emotion. The authors point out that the assumption of a simple face-to-feeling link still shapes legal judgments, security protocols and commercial products.
That matters for any system, human or automated, that reads a face and announces a feeling. It is guessing from a weak signal.
Why is reading a face on video even harder?
Video adds its own distortions before anyone interprets anything. Three show up again and again.
Mirroring. Many video apps flip your own image left to right so it looks like a mirror, and the setting differs from app to app and device to device. If left and right carry meaning in your analysis (and the eye-direction rules above claim they do), you first have to know which way the image was flipped. Often you cannot tell.
Frame rate. Cameras sample motion at different and sometimes changing rates. A blink is fast enough that researchers who wanted to measure one properly recorded at 600 frames per second (Kwon et al., Journal of the Royal Society Interface, 2013). A phone or laptop camera records far fewer frames than that, so short movements get blurred, cut or missed.
Narrow attention. When you focus hard on one thing, you can miss something obvious next to it. In the well-known "gorilla" study, viewers who were counting basketball passes often failed to notice a person in a gorilla suit walking through the scene (Simons and Chabris, Perception, 1999). Staring at a face on a screen does the same thing. You lose the room, the sound and the hands. In my case, I lost the mouse.
Where is face analysis used today?
Face technology is now common at airports, mostly to confirm who someone is. U.S. Customs and Border Protection completed the rollout of biometric facial comparison to all international airports in the United States for arriving travelers in 2022, and runs biometric boarding at select departure locations (CBP, June 2022).
Identity checks and emotion reading are different problems, and the second one is where the science is weakest. Once automation enters, the risks grow: a wrong read happens at scale, and bias in the data becomes bias in the decision. Privacy and ethics questions follow close behind.
How does Okaya avoid the same mistake?
Okaya does not judge a person from one expression. Okaya never identifies who you are from your face. There is no facial recognition.
A check-in is short and private: you talk with Sanora for about 3 minutes. It compares how you sound, what you say and how your expression moves today with your own earlier check-ins. It uses points from your face, not a video of you, and the video never leaves the device.
Three design choices follow from the research above:
- Your earlier check-ins are the main reference. A person who frowns while thinking is compared with their own history, so a habit is not read as a warning sign.
- More than one channel. Voice and words sit next to expression movement, so one ambiguous glance does not decide anything.
- The whole conversation counts. Measures come from the full check-in, so one frame or one glance carries little weight.
The face stays one source of information, read against the same person's past, next to what they say and how they say it.
Key points
- Research does not support reading emotion reliably from a single facial expression (Barrett et al., 2019).
- Eye-direction "lie" rules failed when tested (Wiseman et al., 2012).
- Video adds mirroring, frame-rate and attention problems before anyone interprets a face.
- Okaya compares each person with their own earlier check-ins, across voice, words and expression, and does no facial recognition.
What would have saved me from my mistake?
Context. If I had heard the click a second earlier, I would never have doubted Elvin. Any person or system that reads faces needs the same thing: more than one signal, a baseline for the person in front of it, and the humility to wait for the click.