When Detecting Depression, the Eyes have it

It has been estimated that nearly 300 million people, or about 4% of the global population, are afflicted by some form of depression. But detecting it can be difficult, particularly when those affected don’t (or won't) report negative feelings to friends, family or clinicians.

Now Stevens professor Sang Won Bae is working on several AI-powered smartphone applications and systems that could non-invasively warn us, and others, that we may be becoming depressed.

"Depression is a major challenge," says Bae. "We want to help."

"And since most people in the world today use smartphones daily, this could be a useful detection tool that’s already built and ready to be used."

One system Bae is developing with Stevens doctoral candidate Rahul Islam, called PupilSense, works by constantly taking snapshots and measurements of a smartphone user’s pupils.

"Previous research over the past three decades has repeatedly demonstrated how pupillary reflexes and responses can be correlated to depressive episodes," she explains.

The system accurately calculate pupils’ diameters, as comparing to the surrounding irises of the eyes, from 10-second “burst” photo streams captured while users are opening their phones or accessing certain social media and other apps.

In one early test of the system with 25 volunteers over a four-week period, the system - embedded on those volunteers' smartphones - analyzed approximately 16,000 interactions with phones once pupil-image data were collected. After teaching an AI to differentiate between "normal" responses and abnormal ones, Bae and Islam processed the photo data and compared it with the volunteers' self-reported moods.

The best iteration of PupilSense - one known as TSF, which uses only selected, high-quality data points - proved 76% accurate at flagging times when people did indeed feel depressed. That’s better than the best smartphone-based system currently being developed and tested for detection depression, a platform known as AWARE.

"We will continue to develop this technology now that the concept has been proven," adds Bae, who previously developed smartphone-based systems to predict binge drinking and cannabis use.

The system was first unveiled at the International Conference on Activity and Behavior Computing in Japan in late spring, and the system is now available open-source on the GitHub platform.

Bae and Islam are also developing a second system known as FacePsy that powerfully parses facial expressions for insight into our moods.

"A growing body of psychological studies suggest that depression is characterized by nonverbal signals such as facial muscle movements and head gestures," Bae points out.

FacePsy runs in the background of a phone, taking facial snapshots whenever a phone is opened or commonly used applications are opened. (Importantly, it deletes the facial images themselves almost immediately after analysis, protecting users' privacy.)

"We didn't know exactly which facial gestures or eye movements would correspond with self-reported depression when we started out," Bae explains. "Some of them were expected, and some of them were surprising."

Increased smiling, for instance, appeared in the pilot study to correlate not with happiness but with potential signs of a depressed mood and affect.

"This could be a coping mechanism, for instance people putting on a 'brave face' for themselves and for others when they are actually feeling down," says Bae. "Or it could be an artifact of the study. More research is needed."

Other apparent signals of depression revealed in the early data included fewer facial movements during the morning hours and certain very specific eye- and head-movement patterns. (Yawing, or side-to-side, movements of the head during the morning seemed to be strongly linked to increased depressive symptoms, for instance.)

Interestingly, a higher detection of the eyes being more open during the morning and evening was associated with potential depression, too - suggesting outward expressions of alertness or happiness can sometimes mask depressive feelings beneath.

"Other systems using AI to detect depression require the wearing of a device, or even multiple devices," Bae concludes. "We think this FacePsy pilot study is a great first step toward a compact, inexpensive, easy-to-use diagnostic tool."

Rahul Islam, Sang Won Bae.
FacePsy: An Open-Source Affective Mobile Sensing System - Analyzing Facial Behavior and Head Gesture for Depression Detection in Naturalistic Settings.
Proc. ACM Hum.-Comput. Interact. 8, MHCI, Article 260 (September 2024). doi: 10.1145/3676505

Most Popular Now

Most Advanced Artificial Touch for Brain…

For the first time ever, a complex sense of touch for individuals living with spinal cord injuries is a step closer to reality. A new study published in Science, paves...

Picking the Right Doctor? AI could Help

Years ago, as she sat in waiting rooms, Maytal Saar-Tsechansky began to wonder how people chose a good doctor when they had no way of knowing a doctor's track record...

From Text to Structured Information Secu…

Artificial intelligence (AI) and above all large language models (LLMs), which also form the basis for ChatGPT, are increasingly in demand in hospitals. However, patient data must always be protected...

Deep Learning Model Helps Detect Lung Tu…

A new deep learning model shows promise in detecting and segmenting lung tumors, according to a study published in Radiology, a journal of the Radiological Society of North America (RSNA)...

AI Innovation Unlocks Non-Surgical Way t…

Researchers have developed an artificial intelligence (AI) model to detect the spread of metastatic brain cancer using MRI scans, offering insights into patients’ cancer without aggressive surgery. The proof-of-concept study, co-led...

New Study Reveals AI's Transformati…

Intensive care units (ICUs) face mounting pressure to effectively manage resources while delivering optimal patient care. Groundbreaking research published in the INFORMS journal Information Systems Research highlights how a novel...

One of the Largest Global Surveys of Soc…

As leaders gather for the World Economic Forum Annual Meeting 2025 in Davos, Leaps by Bayer, the impact investing arm of Bayer, and Boston Consulting Group (BCG) announced the launch...

New Computer Models Open Door to Far Mor…

With antibiotic resistance a growing problem, University of Virginia School of Medicine researchers have developed cutting-edge computer models that could give the disease-fighting drugs a laser-like precision to target only...

New Biomarkers to Detect Colorectal Canc…

Machine learning and artificial intelligence (AI) techniques and analysis of large datasets have helped University of Birmingham researchers to discover proteins that have strong predictive potential for colorectal cancer. In a...

Sam Neville Joins the Highland Marketing…

Leading chief nursing information officer Sam Neville is joining the Highland Marketing advisory board. Sam brings a passion for nursing and safety to the board, which debates the big issues...

AI Model Identifies Potential Risk Genes…

Researchers from the Cleveland Clinic Genome Center have successfully applied advanced artificial intelligence (AI) genetics models to Parkinson's disease. Researchers identified genetic factors in progression and FDA-approved drugs that can...

AI Tool that may Assist Underserved Hosp…

As the fields of healthcare and technology increasingly evolve and intersect, researchers are collaborating on the best ways to use emerging technologies such as artificial intelligence (AI) to care for...