Emotional assessment in children with Downs Syndrome using deep learning during Dolphin Assisted Therapy

Abstract

Understanding emotions is critical for supporting behavior and therapy in individuals with cognitive disabilities such as Down syndrome. However, accurately recognizing emotional states in this population remains a challenge due to subtle or atypical facial expressions. This study investigated automatic emotion recognition and assessment in children with Down syndrome undergoing Dolphin-Assisted Therapy (DAT). This work contributes to the development of personalized, data-driven tools to support therapeutic interventions in special needs populations. We proposed three approaches: An image-based approach that involved training raw images on a custom convoluted neural network, a feature-based approach that combined semantic emotion scores from DeepFace with geometric facial landmarks extracted via Mediapipe and combined approach of CNN embeddings and the features extracted. The CNN model trained on raw images only achieved 76% accuracy. The models trained on the feature set performed as follows; Random Forest: 62%, Support Vector Machine :59%, Multi- Layer Perceptron:50% and Dense Neural Networks:50%. Finally, the hybrid model (CNN with Dense fully connected layer) achieved the highest accuracy of 84%. The results show that combining these complementary features improves classification performance, offering a more objective and interpretable method for assessing emotional engagement during therapy.