Floating-Point Transformer 4 – FPT-4

The advent of Large Language Models (LLMs) has generated significant interest due to their demonstrated potential. However, there has been limited exploration from a numerical perspective. While there are existing projects and numerical tools designed to detect numerical anomalies (e.g., InterFLOP, CADNA, VERROU VERIFICARLO, PENE) or to ensure accuracy and robustness (e.g. FLDLib, Fluctuat), these tools still rely heavily on numerical expertise and significant manual effort. The emergence of new formats focused on reducing precision holds promise for enhancing FLOP per watt efficiency, yet there has been limited research into the numerical and performance implications of these formats, aside from discussions about the advantages of random rounding. Although modifications to the accuracy of established algorithms, such as mixed precision in linear algebra routines, have been pursued, the integration of these solutions into various applications remains an uncharted territory.
Recognizing that alterations in representation formats can impact computed results, we propose leveraging generative AI to alleviate the urgent needs to assist HPC developers and numerical experts in adopting low-precision formats for legacy code to reduce computational needs while guaranteeing result accuracy. By analyzing legacy code, we will empower a Large Language Model with numerical knowledge to identify precision reduction opportunities and automatically suggest code modifications. Crucially, we propose to explore how statistical learning could alleviate frugal computing, make mixed-precision algorithms and tools take the extra-small precision format turn, while not sacrificing computational integrity thanks to the use of certified algorithms and numerical tools when needed. This approach empowers developers to embrace efficiency without compromising reliability, fostering sustainable and energy-efficient computing solutions.