Automatic Compiler-based Optimizations for Deep Neural Networks

Deep neural networks (DNNs) are the current state-of-the-art machine learning algorithms in various application domains. Due to their importance, it is crucial that we guarantee their efficient executions on hardware platforms such as GPUs. In this thesis, we optimize the runtime performance and the device memory consumption of DNNs running on modern GPUs. To make our optimizations generic, automatic, and transparent to the frontend machine learning practitioners, we use compiler-based approaches that achieve the optimization objective by carefully analyzing DNNs’ graph representations, tensor expressions, and the hardware platforms on which they run. Compared with manual implementations that require a significant amount of engineering effort, our thesis work, which is made up of three key contributions, requires minimal changes to frontend applications’ source code and can be applied generically to various state-of-the-art DNN models. Our first contribution, Grape, is a new graph compiler on graph-based executions for dynamic DNNs on GPUs. Grape addresses the practicability and efficiency challenges of graph-based executions using three key components: an alias predictor, a metadata compressor, and a predication rewriter. It improves the runtime performance of state-of-the-art text generation and speech recognition workloads by up to 2.99× compared with the machine learning framework baseline, and can optimize workloads that are not practical for prior works on graph-based executions, achieving up to 1.82×speedup over the baseline. Our second contribution, DietCode, is a new tensor program auto-scheduler that efficiently supports dynamic-shape workloads. DietCode addresses the compilation time challenge of auto-scheduling dynamic-shape tensor programs using a shape-generic search space and a micro-kernel cost model. Not only can DietCode reduce the compilation time by up to 5.88× on the state-of-the-art language modeling workload compared with the existing tensor program auto-scheduler, but it also improves the runtime performance by up to 1.70×. Our third contribution, Echo, is a new compiler-based optimization scheme that reduces GPU memory footprint for training state-of-the-art DNNs. We show that by carefully estimating the recomputation's effect on the memory footprint and the runtime overheads, we can significantly reduce the GPU memory footprint by up to 3.13× with only 1% runtime performance overhead, resulting in up to 1.28× faster convergence to the same validation quality. In addition to the system optimization, in Echo, we also build GPU memory profiling tools that accurately pinpoint the GPU memory consumption bottlenecks of DNNs and are integrated into the state-of-the-art machine learning framework.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.