Towards accurate low-bitwidth BERT

Transformer-based models or BERT (Bidirectional Encoder Representations from Transformers) models have seen a lot of success in natural language processing tasks in recent years.However, these networks are very difficult to deploy on edge devices where memory and compute resources are limited.To lower their computational and memory cost, pruning and quantization can be applied.However, naively applying these techniques to compress transformers, can cause a significant reduction in the accuracy of these models.This thesis will focus on improving the performance of quantization-aware training or improving the final performance or accuracy of a quantized transformer-based model without increasing the computational complexity of the forward pass of the network or slightly changing the network's architecture to increase the model capacity.In this thesis, we give one example of such and propose a more efficient quantization-aware training pipeline, called Efficient Two Stage Progressive Quantization in which the model with quantized weights is fine-tuned first, and at the same time, the bitwidth of the weights is lowered.Then, the model with quantized weights and activations will be fine-tuned.At Abstract ii the same time, we strategically choose which bitwidth to fine-tune on and to initialize from, and which bitwidth to fine-tune under augmented data.We are able to ternarize the weights of BERT, compressing it by 14.9x while outperforming the full-precision model performance across nearly all GLUE datasets, and without data augmentation, it can outperform existing BERT ternarization methods.Also, we outperform the state-of-the-art BERT binarization in performance without adding an extra module, saving 18% more memory than BinaryBERT.iii Abrg Les modles bass sur les transformateurs ou modles BERT (Bidirectional Encoder Representations from Transformers) ont connu beaucoup de succs dans les tches de traitement du langage naturel ces dernires annes.Cependant, ces rseaux sont trs difficiles dployer sur des priphriques priphriques o les ressources de mmoire et de calcul sont limites.Pour rduire leur cot de calcul et de mmoire, l'lagage et la quantification peuvent tre appliqus.Cependant, l'application nave de ces techniques pour compresser les transformateurs peut entraner une rduction significative de la prcision de ces modles.Cette thse se concentrera sur l'amlioration des performances de la formation sensible la quantification ou sur l'amlioration des performances finales ou de la prcision d'un modle bas sur un transformateur quantifi sans augmenter la complexit de calcul de la passe avant du rseau ou en modifiant lgrement l'architecture du rseau pour augmenter la capacit du modle.Dans cette thse, nous en donnons un exemple et proposons un pipeline de formation sensible la quantification plus efficace, appel Efficient Two Stage Progressive Quantization Abrg iv dans lequel le modle avec des poids quantifis est affin en premier, et en mme temps, la largeur de bit du les poids sont abaisss.Ensuite, le modle avec les poids quantifis et les activations sera affin.En mme temps, nous choisissons stratgiquement la largeur de bit affiner et partir de laquelle initialiser, et la largeur de bit affiner sous les donnes augmentes.Nous sommes en mesure de ternariser les poids de BERT, en le compressant de 14,9x tout en surpassant les performances du modle de prcision totale sur presque tous les ensembles de donnes GLUE, et sans augmentation des donnes, il peut surpasser les mthodes de ternarisation BERT existantes.De plus, nous surpassons la binarisation BERT de pointe en termes de performances sans ajouter de module supplmentaire, ce qui permet d'conomiser 18% de mmoire en plus que BinaryBERT.v List of Tables xi 3.10 Performance of using Direction Mismatch Distillation (DMD) versus using Mean Square Error (MSE) Distillation. . . . . . . . . . . . . . . . . . . . .

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.