Enhanced multi-modal emotion recognition using the feature level fusion
Bibliographic record
Abstract
Multi-modal human emotion recognition is a complex process of synthesizing information from various modalities to calculate emotion states. This field faces several challenges: (1) Acoustic is an essential component of emotion expression, but it often underperforms compared to visual and text in emotion recognition. (2) Capturing the feature interaction among different modalities is usually complex. (3) Processing high-definition videos can significantly reduce the efficiency of visual analysis. In this study, we presented a learning architecture designed to recognize human emotions effectively. For the first challenge, we implemented a multi-level acoustic encoder (MLAE) that enhances the extraction of acoustic information to improve the acoustic contribution in multi-modal emotion recognition. Facing the second challenge, we introduced the cross-attention block module, which adeptly captures the inter-modal interactions. To address the third challenge, we adopted the re-parameterized visual geometry group network (RepVGG) as the visual feature encoder, employing its multi-branch learning and single-branch reasoning structure to maintain high reasoning efficiency. Our model has demonstrated the state-of-the-art performance of the interactive emotional dyadic motion capture (IEMOCAP) dataset and the multi-modal opinion sentiment and emotion intensity of the Carnegie Mellon University (CMU-MOSEI) dataset.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".