A systematic review and meta-analysis of deep learning approaches for clinical natural language processing: a hybrid transformer framework with prisma 2020 methodology
Bibliographic record
Abstract
Clinical documents, such as discharge summaries, radiology reports, clinical notes and pathology records are the most valuable source of patient health information, but information in them is largely untapped and unstructured for large-scale computational analysis. The use of accurate text extraction, classification and summarisation of clinical text could greatly shorten clinical decision support, pharmacovigilance, clinical trial recruitment and epidemiological surveillance. Transformer architectures and large biomedical corpora pre-trained models have led to significant improvements in clinical NLP benchmarks, like Bio BERT, Clinical BERT and PubMed BERT. Yet there is no systematic review in the field that is quantitative and follows the guidelines of PRISMA 2020 to analyses performance trends over architectures and tasks. In this study, we make three contributions: Firstly, conduct a PRISMA 2020 compliant systematic review and meta-analysis of 312 peer-reviewed studies from 2018 to 2025; Secondly, uncover architectural trends of the past eight years; and Thirdly, propose and empirically test a Hybrid Transformer architecture that uses Clinical BERT for encoding and a GPT-2 clinical decoder to be integrated through a multi-head cross-attention bridge. The results of this meta-analysis clearly show that models based on the RNN architecture (between 64.8% and 65.8% F1 in 2018–2019) are outperformed by those based on the BERT architecture (between 76.1% and 78.3% F1 in 2020–2022) and, in turn, by hybrid transformer models (from 87.8% to 89.7% F1 in 2023–2025). The proposed Hybrid Transformer performs at BLEU-4 = 51.8, ROUGE-1 = 68.4, and F1 = 91.2% for the clinical summarisation benchmark; all of which outperform all the baselines evaluated. The results of the risk of bias assessment by PROBAST showed that 26.9% of studies had a high risk of bias with the highest risk of bias being in the analysis domain. The results confirm the state-of-the-art of hybrid encoders-decoders for clinical NLP and inspire further research on multilingual pre-training and federated learning for privacy-preserving model development in the clinical domain.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.044 | 0.110 |
| Meta-epidemiology (narrow) | 0.003 | 0.002 |
| Meta-epidemiology (broad) | 0.009 | 0.033 |
| Bibliometrics | 0.008 | 0.007 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.004 | 0.003 |
| Open science | 0.003 | 0.003 |
| Research integrity | 0.002 | 0.002 |
| Insufficient payload (model declined to judge) | 0.004 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".