Coreference resolution into machine translation – CREMA

The democratization of large language models (LLMs) in the early years of the project led us to reflect on and question the relevance of the initially envisioned solution for the project's objectives. The development within the LIG of a research axes on the explainability of neural models motivated us to explore model analysis approaches for CA-NMT from this field in order to understand how these models process discursive phenomena during the translation process. There are several research areas within the field of model explainability. "Plausibility" aims to analyze the model's behavior as it is and through the lens of attention mechanism weights to verify its plausibility in relation to a given task; Circuit analysis is an orthogonal research area that provides a more robust theoretical framework.

In order to understand how translation models process discursive phenomena during the translation process, we initially focused on the plausibility approach. From a purely plausibility standpoint, we aim to answer the following question: by analyzing the attention weights of the model from a given sentence to its context, can we say that the model resolves coreference?

To answer this question, we used parallel coreference-annotated data from the ParCorFul2 corpus. We used only the English-German language pair (we encountered problems using the English-French pair). We translated English into German using both the concatenation model and the multi-encoder model described previously, which we will refer to as concat and multienc respectively hereafter.

Then, using neural network-based alignment tools, we aligned the English from the corpus with the English input provided to the models (this is necessary because the models apply tokenization, which makes the text different from the corpus text) and the German from the corpus with the German generated by the models. The alignment process allowed us to identify, within the model texts, the words instantiating the coreferential mentions annotated in the corpus. Modifications to the models enabled us to extract the attentional weights from the sentence to be translated to its context.

Once this procedure was completed, we were able to proceed with the analyses. We conducted both automated quantitative analyses and manual qualitative analyses, involving the participation of six people, four of whom were involved in the MAKE-NMTViz project, which was closely related to this one.

The automated quantitative analyses mentioned above, used to evaluate translation models for their plausibility in resolving coreference during the translation process, are based on three evaluation metrics that we defined to reflect our understanding of how the model should behave with respect to this phenomenon. The first metric, called Max-weight, measures how often the model, when encountering a coreference phenomenon, places maximum attention on the correct antecedent in the context relative to all other words in the context. This metric reflects the ideal situation mentioned earlier: the model only needs context to disambiguate discourse phenomena; for other words, it can do without context. According to this idea, the model should therefore place maximum attention on the coreference and a (very) low weight elsewhere. In practice, for several reasons, some of which are related to technical details of the attention mechanisms' implementation that we won't go into here, this ideal situation is often not achieved. This is why we designed a more permissive metric: the Non-zero-weight. This metric measures the number of times a coreference link to the correct antecedent in the context receives a non-zero attention weight. Although this metric might seem too permissive, since we filter out values ​​below or equal to the threshold representing a perfectly uniform distribution of attention across all words in the context, the fact that a coreference link receives a non-zero weight is already significant. The third metric, called Average-weight, simply measures the average weight given by the model to a coreference link. By definition, the first two metrics are percentages, while the last is an average weight (between 0 and 1).

The two concat and multienc models exhibit very different behaviors with respect to coreference resolution.

For the multienc model, with the three metrics, we have the following values, respectively: 45.91, 88.83, and 0.82.

For the concat model, on the source language side, we have: 10.45, 50.98, and 0.30; on the target language side, we have: 13.25, 33.22, and 0.21.

Notably, the multienc model appears to resolve coreference much better than the concat model. This conclusion is confirmed by the manual qualitative analyses we performed on the same data. We expected the concat model to have a "preference" for placing more emphasis on the target language, as German is an inflected language, and more information is available in the German context for disambiguating coreference phenomena.

Given the work carried out during the project and the considerations mentioned above, we have identified two main avenues for potentially continuing the research envisioned by this project.

The first aims to complement the analyses of the behavior of translation models in context with more detailed analyses, specifically by comparing the two concat and multienc models across several language pairs and under all the same context usage conditions: source language only, target language only, and both. This would allow us to draw more robust conclusions about the models' behavior when facing coreference phenomena during the translation process. Furthermore, we aim for more refined analyses by considering all layers and all heads of attention of the models separately. Some of this work has already been completed or is underway, and, as mentioned previously, it has revealed one head of attention that is more focused on coreference resolution than the others. We could thus achieve the goal of integrating a coreference resolution module into a translation model by leveraging this head of attention. One possible, fairly direct approach would be a two-step learning strategy. In the first step, we learn the model for CA-NMT as we have already done. In the second step, we use the coreference resolution head to allow the model to learn to choose whether context is necessary and which portions (sentences) are actually required.

From the second perspective, we aim to develop an explainability approach based on circuit analysis and inspired by the work of Wang et al. (2022). The idea is to use the same type of approach to identify the model's circuits that implement coreference resolution. Furthermore, a major challenge currently in the field of explainability is identifying invariants for the algorithms implemented by the model's circuits. Indeed, the increasing accessibility of large language models (LLMs) and generative AI raises more and more questions about the interpretation and potentially the control we can exert over the behavior of these models.

Using multiple subsets of data containing different coreference patterns could allow us to identify invariants in the algorithms that perform coreference resolution in translation models. This would be an important result in the field of explainability, and the necessary manipulations of translation models are not specific to these models but could also be applied to LLMs.

In this project we would like to make a step forward in the domain of document-level neural machine translation by dynamically choosing the contextual information that the models uses to generate its translations.
This is opposed to current works where the contextual information is fixed chosen a-priori. The latter solution does not prove very effective even when using a relatively short context. This is due to the fact that, in most cases, the model can translate correctly a sentence without using any context. Words needing a context for their correct translation are relatively rare, thus learning specific model contextual parameters for taking them into account is difficult, as the training signal is sparse.
In this project we would like to study models with a more compact context. The choice of such context is leaded however by an additional module which is able to detect the most ambiguous words a translation model can face: discourse phenomena, and in particular anaphora and coreferences.
Another aspect we would like to study in the project is the specificity of the evaluation of Document-Level Neural Machine Translation models.
Indeed for such models, the BLEU evaluation metric is not adapted. The words needing a context for their correct translation are relatively rare, their impact on an automatic evaluation metric like BLEU is thus limited. Their correct translation however, and more in general contextualized translation, has a non negligible effect on the translation quality as perceived by a reader.
For a better evaluation of contextual models, contrastive test suites have been designed. We find that such kind of evaluations can be improved by using more realistic sentences. Current test suites contain indeed mostly artificial sentences choosen ad-hoc.
The main objectives of the CREMA project (Coreference REsolution into MAchine translation) are: 1) designing new models for coreference resolution; 2) integrating a coreference module into Document-Level NMT models so that to allow a dynamic context choice, based on ambigous discourse phenomena detected by such module; 3) designing a new test suite, more effective for Document-Level NMT models evaluation.