Between 2022 and 2023 I developed a real-time emotion-recognition pipeline for children with autism. It pairs facial landmarks with a convolutional neural network and runs on a Jetson Orin Nano, a small computer that can sit right next to the camera.
The data came first
I combined image and video data from CALMED, Kaggle, and other sources and cleaned it into one training set. CALMED is the one built for this problem: a dataset for emotion detection in children with autism. Combining sources like these is most of the work. They label emotions in their own ways and record faces in different conditions, and they have to be cleaned into something one model can learn from.
With the data combined and cleaned, the pipeline reached about 92% development accuracy.
Running it live
The Jetson Orin Nano runs the pipeline in real time: facial landmarks describe the shape of the face, and the CNN classifies the expression. The code for the network is public. It trains a small Keras CNN on 48×48 grayscale face crops, runs it on a live camera feed, and evaluates it with a confusion matrix and a classification report. The dataset itself isn’t included.
What the output means
A facial expression is not a direct reading of what someone feels, and about 92% on development data says how the model does on that data, not how it will do with one child in one room. So I built it as a support tool: a signal that can help the people around a child, never a diagnosis.
Afterwards
The prototype and the concept were later acquired following a Gulf-region competition.