Systematic Exploration of Power-Augmented Feedforward Layers in Transformer Networks

Multi-layer perceptrons (MLPs) represent the foundational architecture of deep learning neural networks, consisting of fully-connected layers organized in a hierarchical structure. While activation functions between layers introduce non-linearity to the overall network, the fundamental computation within each layer remains strictly linear, where each neuron computes a weighted sum of its inputs. This linear nature of individual layer operations is a defining characteristic that constrains the expressiveness of each layer to linear transformations, with the network’s overall representational capacity determined by its depth (number of layers) and width (number of neurons per layer). In this thesis, the efficiency of using power-augmented networks in transformer models will be investigated. Power-augmented networks utilize trainable powers on their connections, allowing powers to be any real numbers. This introduces a more expressive form of learning non-linearity in neural networks and enables the learning of complex curves using non-linear power-augmented connections potentially enabling capturing of intricate patterns more efficiently. While conventional networks are fundamentally limited to learning piecewise linear functions, power- augmented functions can capture complex curves with fewer resources than traditional approaches. A crit- ical challenge addressed in this work involved ensuring numerical stability, particularly when dealing with fractional powers and small input values. Through systematic experiments, this thesis delivers an optimized NVIDIA CUDA implementation for the power-augmented perceptron layer and novel solutions to training stability challenges inherent in networks with dynamic power parameters. The presented evaluation framework demonstrates that power-augmented networks can achieve optimal performance with significant trade-offs across measurable metrics such as parameter count, validation loss, training time, and inference time. Key findings show that power-augmented networks can achieve superior or comparable validation loss performance to deeper baseline models while requiring fewer parameters and offering reductions in computation time and latency.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.