Large generative mRNA language foundation model for efficient coding sequence generation and design with mRNA-GPT
Bibliographic record
Abstract
Abstract mRNA design plays a central role in synthetic biology, nucleic acid therapeutics, and vaccine development. Although large language models are applied in many biological fields, generative language models for de novo mRNA design remains largely unexplored. Here, we introduce mRNA-GPT, a series of generative mRNA language models which for the first time covers the three domains of life as pretraining datasets. Based on a GPT-2 transformer architecture with 302 million parameters, we pre-trained three separate models on 19,676 bacterial, 4,688 eukaryotic, and 702 archaeal species, leveraging 80 million, 83 million, and 2 million mRNA coding sequences, respectively. Distinct clustering of mRNA coding sequence embeddings from animals, plants, and fungi in the pretrained mRNA-GPT-eukaryote indicates that the model captures organism-specific sequence features. Following unsupervised pre-training, we fine-tuned mRNA-GPT on a translation-efficiency dataset to generate high-performance mRNA sequence. Compared to the pretrained model, the fine-tuned mRNA-GPT produced mRNA sequences with significantly higher translation efficiency scores, demonstrating the ability of mRNA-GPT to capture sequence features underlying high translation efficiency. We further fine-tuned mRNA-GPT on datasets for mRNA stability and mRNA expression, where it likewise produced high-performance mRNA sequences. Our pretrained models are publicly available, enabling other researchers to adapt mRNA-GPT to specialized tasks such as tissue-specific mRNA expression or stability by fine-tuning on their own data. Together, our study demonstrates that generative mRNA language modeling as a promising approach for accelerating mRNA design across diverse biological fields.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.002 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.004 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".