Multimodal Emotion Recognition Using Deep Learning for Audio and Visual Fusion
Bibliographic record
Abstract
The study of emotion recognition leads to significant advances in affective computing, human-computer interaction and artificial intelligence. Traditional unimodal methods, which uses either facial expression or speech signal, do not give high accuracy due to various factors such as noise, occlusion or the limitations of one of the methods. To solve these problems, the proposed work introduces a multimodal emotion recognition framework that can identify human emotion through facial expression as well as speech-based emotion. The framework uses deep learning models for feature extractions. Convolutional Neural Networks that capture temporal features of facial expressions from the facial images and spectrogram-based CNN Long Short Term Memory model that deals with audio signal to get spectral features of speech. The results of audio and visual methods are combined into multimodal fusion layer followed by the classification into emotion. The work aims to improve on unimodal systems by combining both audio and visual modalities. The proposed multimodal framework has an overall accuracy of 88.6% and a macro F1-score of 87.9%. This is better than unimodal facial and speech-based systems by 6.4% and 4.8%, respectively. These quantitative findings validate enhanced robustness and reliability relative to single-modality methodologies, especially in noisy and real-world contexts. Some of the benefits that are expected to arise from the current study include more accurate and robust recognition in real-life situations as well as some interesting application area such as healthcare, e-learning and intelligent conversational agents.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.001 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.001 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".