Generative modeling, Heavy tails, Outliers, Sparse Training – GHOST

Generative Artificial Intelligence (GAI) models are expensive, with massive energy requirements for both training and inference (use in applications). As GAI models are increasingly adopted to solve problems across industry, significant changes in how we train and use these models are required both to realize carbon emission goals, and democratize access to GAI models and research. State-of-the-art approaches for compressing neural networks are of limited efficacy when used with GAI models. While in most neural networks 85-95% of the weights can be pruned while maintaining performance, GAI cannot be pruned beyond ~70% sparsity without significant degradation in performance. Empirically it has been observed that GAI models have different training dynamics that are likely responsible for affecting their compressibility: (a) trained GAI models have outlier weights/activations that appear to be important, and render conventional pruning and quantization less effective, (b) it appears that lower-magnitude weights carry more importance in GAI models than other deep learning models. Both of these empirical observations are currently poorly understood. Recently, we have illustrated that such outliers in optimization may occur due to the emergence of “heavy tails”, and heavy-tailed distributions have tight links with compressibility. In this proposal, our main objective is to develop a theoretically sound algorithmic framework for achieving state-of-the-art compression techniques for GAI. We will first explore the connections between heavy-tails and the behavior of the outliers observed in GAI, and understand how the training dynamics of GAI differ from other deep learning models. By exploiting this connection, we will then develop efficient algorithms that will significantly reduce the computational complexity both in memory and run-time. We will produce open-source software and test their performance on applications on computer vision, audio/language processing.