Current opinions on large cellular models

Large language models (LLMs) have made breakthroughs in natural language processing (NLP) and understanding, and have brought revolutions in many other fields [1-4]. Inspired by those successes, several large cellular models (LCMs) adopting similar structures of LLMs have been developed for single-cell transcriptomics, including (but not limited to) scBERT [5], Geneformer [6], scGPT [7], scFoundation [8], and GeneCompass [9]. The practices of these models have shown LCMs’ power and potential in various biological tasks and illustrated the possibilities of revolutionizing future biological studies by LCMs. Figure 1 shows a general framework of LCMs. The models are pretrained on massive single-cell transcriptomic datasets of multiple human tissues and organs or even including data of other species. Each cell in the dataset is characterized by gene expression values and serves as a training sample. Similar to LLMs, LCMs commonly use the transformer [10] architecture and are generally pretrained using a self-supervised strategy of either masked modeling [11] or autoregressive generative modeling [12] techniques. In masked modeling, we first randomly mask some genes’ expression in a cell, and then let the model convert the genes and their expression values into embeddings, and predict the masked gene expressions based on observed genes as the contexts. The learnable parameters in LCMs are optimized to minimize the loss function between predicted and ground truth gene expression values. Usually, the sizes of the models and the numbers of parameters are very large, currently at levels from tens of millions to billions. That is why these models are called large models. It has been shown that given sufficient data, the models can benefit from larger sizes in their pretraining performances on both the natural language data [1] and single cell data [8]. The pretraining procedure enables LCMs to learn about gene relationships as well as cell characteristics. After pretraining, LCMs can be applied in both zero-shot and fine-tuning paradigms, solving an array of downstream biological tasks including cell annotation, gene network inference, drug target discovery, in-silico perturbation, etc. Schematic diagram of large cellular models. Pretraining: Pretraining is a process where a model learns from large amounts of data before being used on specific tasks. This initial phase allows the model to capture general patterns and semantics, enhancing its performance and adaptability across various specific applications. Self-supervised learning: Self-supervised learning is a strategy of machine learning that does not use extra labels to supervise the learning procedure, but instead uses the task of predicting certain parts of the data from other parts as the objective of learning. This approach eliminates the need for labeled data, as the model generates its own supervision signal during training. Embedding: Embedding in the general sense is to represent objects such as text, images, genes, or amino acids using mathematical vectors in certain continuous space where the locations of the vectors represent meanings or semantics of the objects with regard to their relations. In LCMs, embeddings are compact representations of the transcriptomic data in a mathematical space, encoding gene symbols and expression value information. LCMs can also obtain embeddings for cells and cell states. Loss function: Loss function is one of the key elements in the design of a machine learning model and algorithm. It quantifies the difference between predicted and true values, guiding the model to adjust its parameters to minimize this difference. Transformer: Transformer is a neural network architecture that uses self-attention mechanisms to capture relationships between embeddings of the input objects (tokens). It is widely used in large artificial intelligence (AI) models nowadays due to its scalability and effectiveness. Masked modeling: In masked modeling, parts of the input data are replaced by a specific mask embedding, and the model learns to predict these parts based on the remaining information. It is like a “cloze test”: When the model learns to predict the masked data, it “understands” the data. Autoregressive generative modeling: In autoregressive generative modeling, the model generates new data points given the preceding ones in a sequence. In LCMs, gene expression rank can be regarded as sequential information. Zero-shot learning: Zero-shot learning is a machine-learning paradigm where a model is trained to perform tasks it has not been explicitly trained on. In LCMs, it usually refers to extracting cell or gene embeddings for downstream tasks from the pretrained model without the need for extra training for specific tasks. Fine-tuning: Fine-tuning is a process where a pretrained model is further trained on a specific task or dataset to improve its performance. This is particularly useful when the pretrained model has learned useful representations that can be adapted to the target task, saving time and computational resources. Large AI models in biological fields such as LCMs on single-cell transcriptomics data have begun to flourish in the past 2 years. At this critical time for great advancement and rapid development, Quantitative Biology took the honor to interview some of the key authors of the most recent representative works of LCM. The authors we interviewed and the models in the order of their publication or preprint release are: Fan Yang and Jianhua Yao (FY&JY) of scBERT [5], Christina V. Theodoris (CT) of Geneformer [6], Bo Wang (BW) of scGPT [7], Xuegong Zhang (XZ) of scFoundation [8], and Xin Li and Ge Yang (XL&GY) of GeneCompass [9]. We asked their opinions on a few common questions and questions specific to their models. The questions range from brief introductions to their models, the advantages and challenges, characteristic biological applications, and their perspectives for the future of the field. We collect these interviews upon all authors’ consent to form this special article to reflect the current opinions on this exciting topic. These opinions shed light on the transformative impact of LCMs on biological research and provide a glimpse into a future where AI and life sciences converge to answer the key questions about life. FY&JY: scBERT [5] is a pretrained language model designed to transform single-cell transcriptomic data into a universal embedding. This transformer-based model is trained using the BERT [11] paradigm and could be utilized for various applications, including cell type annotation, discovery of novel cell types, and identification of novel marker genes. CT: Geneformer [6] is a foundational deep learning model pretrained on a large-scale corpus of ∼30 million single-cell transcriptomes to enable context-specific predictions in settings with limited data in network biology through transfer learning. With both zero-shot learning and fine-tuning with limited data, Geneformer consistently boosted predictive accuracy in a diverse panel of biologically meaningful downstream tasks. We demonstrated Geneformer’s ability to make new biological insights including discovering a novel transcription factor in cardiomyocytes through zero-shot learning and revealing candidate therapeutic targets for cardiomyopathy using limited patient data, both of which we experimentally validated with functional assays in cells. Geneformer’s fundamental understanding of gene network dynamics can now be democratized to a vast array of downstream tasks to accelerate discovery of key network and candidate therapeutic scGPT is a single-cell model pretrained on million human cells. Inspired by LLMs, as a to first learn single-cell biology from the large-scale diverse pretraining dataset of human and then transfer the to a of downstream tasks. scGPT a transformer with to capture the between genes. scGPT a special with cell and gene that enable generative training with the data in an autoregressive The pretrained scGPT model to new cell in zero-shot and with gene its is to various consistently models in cell type and tasks. scFoundation is a pretrained model on single-cell transcriptomics data and is on a large in of gene and of training data. from the model architecture design and training it can provide embeddings for both single-cell and These embeddings can be applied to a of tasks such as enhancing gene cell types, and predicting drug in tissues and cells. scFoundation can embeddings useful for gene and predicting the of on single cells. GeneCompass is a pretrained on million human and single-cell of that gene gene and are into GeneCompass by encoding into the GeneCompass can a range of across the biological including cell type annotation, drug and GeneCompass can accelerate the discovery of critical cell by in gene in the scBERT the first model developed for single-cell about the and in the FY&JY: We scBERT in and the and preprint in scBERT is the first model for single-cell data. Inspired by the of BERT [11] paradigm in we to BERT on single-cell data. This several challenges, with the one being the of single-cell data in into input for the After a of the of and with we designed gene and expression embedding, to and in The of these embeddings has the design of several transformer-based models. a large of training data and to self-supervised are the key for AI models to be or for life FY&JY: the key for AI to be or for life tasks in the and a biologically into an by AI is to a model based on data. it is a common in life scBERT on the research in FY&JY: We have AI for and single-cell In the of we developed a cell type and a for single-cell we developed a general framework and a of works on the and its potential biological applications. is on the potential of AI models in biological FY&JY: AI could various of biological from understanding and discovery to single-cell data allows to the between and in cells from a this data other to the such as can process and new targets for novel modeling between and other we can design novel In we can use AI to the of and improve the of life. The fields of both AI and biology are very with or even in an is on the recent future of AI in biological FY&JY: we all that has the biology by predicting from sequence. models have been to predict the of where with other such as and In the recent AI for with cellular by single-cell is to for the biological the biological from of Geneformer on biological those also be with or machine-learning CT: We demonstrated Geneformer’s ability to enable predictions in a diverse range of biological settings with both zero-shot learning and including in gene network dynamics predicting gene network transcription factor targets and transcription factor dynamics predicting and transcription factor cell in and and genes and candidate therapeutic We applied Geneformer to a range of and to the of its fundamental We also Geneformer to machine learning such as and and that Geneformer consistently boosted predictive In of novel we designed an in approach that a novel transcription factor in cardiomyocytes through zero-shot learning that we experimentally critical to the ability of the cells to It exciting to that Geneformer to this novel of research on cardiomyocytes and that the predictions validated to have a true biological in cells. We then approach to an in which novel therapeutic targets in cardiomyocytes that the ability of the cells to in an cell model of We that the model to predict novel therapeutic targets that a true biological impact on the in cells and are to use Geneformer to in other and biological settings in the are a on can some in research that AI LCMs make CT: one of the in biology is that it is and to all of the of one could in to network and therapeutic of the of AI is the ability to downstream in an by a data from the downstream in the can provide to the model about where its predictions and where the predictions with data. In the of predicted that the of transcriptomic data to future models pretrained on even to meaningful predictions in even tasks with limited can some on task that into this CT: We pretrained Geneformer in and then has been a rapid in the and of single-cell transcriptomic data in the of the most tasks we in the Geneformer predicting transcription in or range with their This is an task for the model to predict using transcriptomic data without about the model to predict to a this of transcription machine learning models are pretrained on even amounts of data, further fundamental that allows to predict these of genes. Geneformer to predict network of genes with as few as cells. the model further fundamental during pretraining, these tasks could be with even cells or even by zero-shot learning with fine-tuning data. does in AI models for CT: The to be the to sufficient to the of models we are which is most settings are to to in is data that is in the with and many to about the data or such as the from or etc. we the of AI in it be to to data as to the most of the in biological The is a of this that allows to large amounts of single-cell data, an for to for other of biological data. It has been 1 can brief to the research in that is on or with Geneformer or other large cell many are to AI large models into their have to on the to make their need to be CT: AI and to in gene network of is on new AI models transfer learning approach to new such as gene cell through space and The other of is these models to questions in gene and to for human We between computational and to a AI and strategy to accelerate In of AI into biological one common is that to all of their tasks of is very and can be the difference between a model learning at all or a predictive that data are between other potential and data to into and by as to cells from all for all a is used to a is used to the of the model to data. the fields of AI and of biology are very with or even in an is on the recent future of AI in biological and CT: This is an exciting time in the fields of both AI and With the current of large-scale biological data, we are now the where large-scale AI models can be trained to have a fundamental understanding of model are by the biological research is also an to in AI based on the of biological such as the need to by the of that the structures that are etc. have been on the of large cell models in biological but are also on the of large models. are that the performance of LCMs on many single-cell tasks can also be by designed for those specific but those can be in of the data and is on such It is without a that models can be optimized to perform well on specific datasets for specific tasks. are with the approach that LCMs can the modeling of models is limited by of this most of the current on gene or other to the input and the other LCMs use to capture from the a of gene that could from biological are models to datasets or and are to to from the at the model learn cell representations from large-scale data biological from a of LCMs are all based on the transformer developed for tasks in LCMs used to the to be on biological data. on in are the key in adopting LLMs for that is a need or to design structures for biological data and The key that we when scGPT to generative pretraining used in LLMs to LCMs, given the of single-cell data. a autoregressive training and with the of a of gene and predicting cellular to BERT and models, which are also without are that with the of large models, the for research has with the when most research can be with mathematical models and based on data. on this or have for with on benefit from the advancement of current We to that benefit the and in their scGPT we the model on scGPT where can the model with a dataset It is to when new models, and we with to as we the the fields of AI and of biology are very with or even in an is on the recent future of AI in biological and AI is a that modeling power to to model the in predictive models in research have the in and the of patient We AI to be in a similar for biological with in the to their and are some in the that LCMs are of to single-cell data. on are the key in a model that is for cell are the key that scFoundation from other The key of the model in the and characteristic of data, and in from biological when modeling cell as a and gene expression value as a the genes make the a that to for data across and in these challenges, scFoundation the transformer-based architecture and a novel pretraining task that on masked language modeling The architecture an that continuous gene expression values into learnable vectors without and an to learn the relationships between genes the of single-cell gene expression data. In modeling, the task is to predict the masked gene expression in a cell using the by other genes the cell, those gene expressions are with the or This approach not relationships but also cells across a design of scFoundation other LCMs. are to LCMs into their are the most for or to use scFoundation in their are the from such the of LCMs such as scFoundation into biological it is to their into tasks and tasks. tasks on of such as cell type or drug tasks are about understanding relationships between genes or predicting in gene such as through gene network or gene a common approach with scFoundation is to the to obtain These embeddings can be and applied to various downstream representations of cells for in without the need for computational or This process the of embeddings from downstream in the of models. the extracting gene embeddings from the scFoundation embeddings for all genes cell, the of gene This is for to and these embeddings can as to models such as enhancing the accuracy of their This approach allows to scFoundation to their research both in of and are that with the of large models, the for research has with the when most research can be with mathematical models and based on data. on this or have for with on benefit from the advancement of current We the the between LCMs and we that these are not but in scFoundation we demonstrated that the scFoundation model in performance when for specific it can also be with models to performance. This a future for large-scale models into the computational of training. with limited computational we for both the model and as by scFoundation and other LCMs on we have developed an and an for which allows to pretrained embeddings for tasks. This a and for the need for or In the that observed a that the performance of the model when data and larger models are observed the on a to the model we have observed a similar to that in in We pretrained scFoundation models with and million and that the to predict masked cell gene expression with This that even larger scFoundation models could even predictive that we have not the of the model the model several as many single-cell datasets as is for models in the that larger datasets performance. the model architecture is given the loss and data in LCMs to it is to design pretraining models that the these are the for LCMs can be by models trained with a similar of data also be we are at the of development, it is to the for their In the of we to model it such as being on a single and on commonly such as the This approach to a between computational power and that the scFoundation all genes for could the model be to This a to insights into cellular biological data can a on cellular states. the of scFoundation to data, such as we could several approach tasks that both gene expression and data. we could models to predict gene expression values based on and when predicting we could the scFoundation architecture by transformer for gene from the pretrained These embeddings then be by the new to predict the with specific gene the vast of it be to transformer to the data. The is the rapid advancement in the AI is the of such as images, and these also be in biological learning is one paradigm in current large AI models and has shown great in many tasks. by the of we have developed a new model that uses the generative paradigm to learn the cell language that both gene expression data and various of have shown its in certain tasks and its power in of single-cell transcriptomics data that can biological information. It has shown potential for cellular such as in and of cells a of biological and of data multiple biological is a that many are on. We that the advancement in data understanding and for provide or for it is to predict this fundamental difference between the biological and the is that are in understanding data and their but are from understanding the biological data and their meanings in many It is a to but that are with current of data and is of GeneCompass to other LCMs is that it trained with data from both human and is the benefit of this downstream tasks are that can be with GeneCompass but not LCMs pretrained on human in the massive single-cell data is to a LCM. with and larger is to a performance. with LCMs pretrained on human data, GeneCompass is the one that pretrained on the most of data, million which made it some LCMs on several downstream tasks. The biological is the between human and The learned gene of a be to the other one through gene which be of for those downstream tasks that data. on human data, cell type is one of the downstream tasks that be on pretrained we utilized cell type as a to human cells on datasets from organs and to is observed in the GeneCompass can with and even the cell new of GeneCompass is the of in modeling the genes. that data and in AI is the future for biological tasks. on the or on this based on of the LCMs are by data through a self-supervised learning the can the life that training data not The strategy that with its we of including gene and by encoding into an and with the of the single-cell This is a strategy of at the input are also some other such as pretraining task and The strategy an of research and be the of future are the key in the of the GeneCompass model and its pretraining when made the model for The key for a is to single cell data from human and genes are from their gene be a and but of the gene The gene computational power and this we designed a strategy that genes between human and based on the The genes the gene in gene we the human and into a space to across species. of the gene with ones the of this has of key transcription most single-cell data are of the gene expression at the time when the is LCMs in understanding cellular such as and key that such the LLMs in the general the of pretrained LCMs is that training on diverse datasets from a range of the model learn a understanding of cell, and life. most single-cell data are the range of pretrained data the of cell on the pretrained LCMs, similar cell data have in the space, which in understanding cellular the key we in gene to or a gene to a certain expression the of the cells with the cells and the target the potential key can be The from and Geneformer both the of this also the function of the key This is of great for the discovery of critical cell and candidate drug the fields of AI and of biology are very with or even in an is on the recent future of AI in biological and the between AI and life breakthroughs in biological and with the of models, LCMs can well to tasks without the need for specific training or with for a range of downstream applications. one the use of LCMs the time and with biological and and in and in be through LCMs, which improve the of tasks such as drug discovery, etc. the other have AI models at levels of life such as the LCMs at the and at the and at the also have been LLMs that can be used to data such as that be a model to biology and model the to the relationships between and of all This interview the current opinions on the of LCMs and their from some of the in this field. The models we interviewed on have shown great in many downstream biological but the is in its It is to the about be the or model for single-cell transcriptomics data. The authors’ insights on the of models, which can to obtain an understanding of the of models. It is that the authors their and for the key in and adopting transformer-based AI models for biological such as single-cell data into and and model performance with limited data and resources. also in model and including the of general pretraining to single-cell data and the design of models that can biological from The opinions also and practices on the in computational for downstream and for the of to make large AI models to The authors also their perspectives and for using LCMs in future biological to the of AI in and the of biology and for the of machine learning Each its own data and design their own to their models and and to of their This is not for the of the field. Christina V. Theodoris this and on it in a article in this This is an that can have in the of AI for biological tasks. Biology is and we for that can answer all biological questions even the has been in many other a at multiple levels for and that the of biological questions is a in on AI for life Xuegong Fan Jianhua Christina V. Bo Xin Ge The authors the that have their based on which the the of and the of The authors of is not to this article as new data or in this

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.