Bias, fairness and ethics of compressed NLP models – DIKE

Natural Language Processing has been moved a step forward with the advent of the Transformer architecture in 2017, allowing parallel training on GPU. This has led to very large language models (BERT being the most popular). However, these models are memory hungry. Neural weights compressing techniques have been proposed: weights quantization, weights pruning and knowledge distillation. These methods are all very close to maintain the same level of accuracy of the original model with massive memory savings. Beyond accuracy, we will study in the Diké project how model compression techniques affect model biases, fairness and ethical abilities of existing compression techniques (there is no free lunch). We will also propose new compression algorithms that will prevent, by design, bias, fairness or ethical issues in representations or predictions.