Up-to-date LLM for all – LLM4all

Large Language Models (LLM) of sufficient size exhibit outstanding emergent abilities, such as learning from their input context and decomposing a complex problem into a chain of simpler steps. Both these emergent abilities and the performances that have been published on many tasks tend to prove that the model size is an important feature to obtain the best possible generic LLMs. The LLM4all project will thus focus on such large models, or on models at the same level of generic performances, and will propose methods to solve two related fundamental issues: how to update these LLMs automatically, and how to reduce their computing requirements in order to facilitate their deployment. It is indeed not sufficient to simply continue the LLM pretraining on newest data to solve the former issue, because of catastrophic forgetting, i.e. the model losing some of its previous knowledge and abilities. A much heavier retraining on both old and new data is thus required, which is often too costly and justifies why even commercial models such as GPT-3 only contain information that is dated pre-2021 (as of April 2023). We will thus propose new solutions to this problem, for instance by combining growing neural networks with sparse architectures, and we will distribute multilingual LLMs that have been automatically updated, based on for instance BloomZ and Whisper. With regard to the latter issue, about reducing the costs of processing these large LLMs, we will propose several solutions, depending on the target use case: either by adapting to a specific model and hardware algorithmic optimisations that enable to trade speed for memory; or through voluntary collaborative computing that distributes the computation load onto several nodes; or with sparse approaches such as Mixture of Experts, which are well adapted to growing neural networks; or with distillation methods when the target task is known. Beyond releasing generic LLMs, these approaches will also be validated on two use cases in French: automatic meeting summarization and emergency calls analysis. For the first use case, an up-to-date LLM will be trained on a large available dataset of meetings and will be released in open source. For the second use case, an up-to-date LLM will be adapted to a corpus of simulated Emergency calls and combined with external information sources, including speech and medical ontologies.