The Cure or the Curse: Investigating the Role and Challenges of Positional Encoding in Transformers
Bibliographic record
Abstract
The Transformer architecture has been the driving force behind large language models (LLMs), enabling us to leverage immense computational power and data.However, it is essential to consider the parallel nature of Transformers and the specific mechanisms at work within them to help representing sequential data-particularly positional encoding.This adhoc component has become de facto standard in contemporary models without much clarity on its role.This thesis investigates positional encoding by introducing novel methodologies to expose its implications and uncover its potential shortcomings.First, we scrutinize Absolute Positional Embeddings (APEs), used in LLMs like GPT-3, which represents order by encoding each word's absolute position.Specifically, we assess APE's ability to encode sentences across all positions in Chapter 3. Our analysis reveals disparities in representation when sequences start at different positions, questioning APEs' efficacy in enabling Transformers to encode relative positions.These observations indicate APEs may cause overfitting on positions, impeding performance.Hence, we explore eliminating positional encoding in decoder-only Transformers in Chapter 4. First, we provide mathematical proof that decoder-only Transformers is capable of recovering positional encoding.Furthermore, we show it performs on-par or better than explicit positional encoding methods on length generalization in downstream tasks.This research scrutinizes the exact role of positional encoding in Transformers, highlighting the drawbacks of widely used methods and casting new light on our understanding of these models.Our findings reveal that Transformers without positional encodings can function effectively, and even show improved performance in certain tasks.Such findings pave the way for developing more optimized positional encoding techniques in future Transformer models.i My co-author, Koustuv Sinha, deserves a special mention for his impact on me about research and productivity, as well as his assistance in advancing my career.Similarly, my gratitude extends
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".