L'intelligence artificielle pour la classification automatisée des textes courts issus de la vigie psychosociale : des méthodes classiques aux grands modèles de langage
Bibliographic record
Abstract
RÉSUMÉ : Cette étude explore différentes approches pour automatiser la classification et l'analyse des préoccupations psychosociales exprimées par la population d'une région du Québec. Ces préoccupations sont recueillies sous forme de textes courts dans le cadre d'une veille psychosociale mise en œuvre pendant la pandémie de COVID-19 et qui se poursuit. L'objectif est de concevoir une application web permettant d'assister les intervenants en santé publique dans l'automatisation de la gestion de données. Trois études ont été menées. Tout d'abord, une analyse comparative des algorithmes traditionnels de machine learning (k-NN, SVM, XGBoost) et des modèles de deep learning basés sur les transformateurs, combinés au Finetuning et au SetFit, a montré que les approches utilisant des sentence-transformers sont les plus performantes, atteignant une précision de 70,74 %, contre 68,69 % pour le modèle SVM. Ensuite, une modélisation thématique appliquée à un corpus d'environ 2000 entrées collectées dans les enquêtes mensuelles de la Vigie a permis d'identifier des tendances psychosociales émergentes. La méthode LDA avec unigrams a obtenu les meilleurs résultats (cohérence de 0,59). Enfin, des grands modèles de langage (LLMs), comme LLama3.1, LLama2 et Mistral, ont été intégrés pour transformer la classification en une tâche de génération de texte explicatif. Cette méthode a permis une classification multi-étiquettes et la génération d'explications adaptées aux besoins des équipes en santé publique, avec une précision atteignant 97.2 %. -- Mot(s) clé(s) en français : Intelligence artificielle, classification automatique, modélisation thématique, grands modèles de langage, préoccupations psychosociales, génération adaptative. -- ABSTRACT : This study explores different approaches to automating the classification and analysis of psychosocial concerns expressed by the population of a region in Québec. These concerns are collected in the form of short texts as part of a psychosocial surveillance effort launched during the COVID-19 pandemic and currently continuing. The aim is to design an web application to assist healthcare actors in automating data management. Three studies were carried out. Firstly, a comparative analysis of traditional machine learning algorithms (k-NN, SVM, XGBoost) and deep learning models based on transformers, combined with Finetuning and SetFit, showed that approaches using sentence-transformers performed best, achieving an accuracy of 70.74%, compared with 68.69% for the SVM model. Next, thematic modeling applied to a corpus of around 2000 entries collected in the monthly Vigie surveys identified emerging psychosocial trends. The LDA method with unigrams obtained the best results (consistency of 0.59). Finally, massive language models (LLMs), such as LLama3.1, LLama2 and Mistral, were integrated to transform classification into an explanatory text generation task. This method enabled multilabel classification and the generation of explanations tailored to the needs of professionals, with an accuracy of up to 97%. -- Mot(s) clé(s) en anglais : Artificial intelligence, automated classification, topic modeling, large language models, psychosocial concerns, adaptive generation.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.004 | 0.010 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.003 | 0.002 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.004 | 0.002 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.002 | 0.003 |
| Insufficient payload (model declined to judge) | 0.005 | 0.004 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".