Kaligo-based Intelligent Handwriting Teacher – KIHT

Handwriting with digital pens is a common way to facilitate human-computer interaction through the use of Online Handwriting (OH) trajectory reconstruction. We use the digital pen of Stabilo “DigiPen” equipped with sensors from which we wants to reconstruct the OH trajectory. Such a pen allows to write on any surface and to get the digital trace, which can help learning to write, by writing on paper. We design a novel processing pipeline that maps the sensor signals of the pen to the corresponding OH trajectory. Notably, in order to tackle the difference of sampling rates between the pen and the tablet (which provides ground truth information), our preprocessing pipeline relies on Dynamic Time Warping (DTW) to align the signals. We introduce a dedicated neural network architecture, inspired by a Temporal Convolutional Network (TCN), to reconstruct the online trajectory from the pen sensor signals.

During the first year of the project, IRISA's work focused on the pre-processing chain for processing, synchronizing and labeling data (sensor and trajectory) to make it usable for deep neural network learning. In addition, several experiments were carried out on different types of deep neural network architectures (CNN, TCN...).
During this second year, we collected a significant amount of data using the latest generation Digipen. The pre-processing chain was finalized. It is based on automatic alignment between ground truth (trajectory captured on digital tablet) and DigiPen sensor data (kinematic sensor) using the Dynamic Time Warping (DTW) algorithm.

A neural network architecture based on Temporal Convolutional Networks (TCN) was designed and its hyperparameters optimized. A learning strategy based on on touching movements (surface contact trajectory) was implemented. For optimal analysis of the results, an evaluation based on Fréchet distance was implemented.

These contributions have been published [1] in the International Journal on Document Analysis and Recognition (IJDAR) in March 2023, and will be presented at the International Conference on Document Analysis and Recognition (ICDAR 2023) in August. In addition, following this publication, we have made some of the data freely available to serve as a benchmark for further research.

[1] Wassim Swaileh, Florent Imbert, Yann Soullard, Romain Tavenard, Eric Anquetil. Online Handwriting Trajectory Reconstruction from Kinematic Sensors using Temporal Convolutional Network. International Journal on Document Analysis and Recognition IJDAR (2023), 2023. ?hal-04076399v2?

The main challenges currently being studied are :
(i) modeling untracked / complex hovering trajectories (trajectory above the surface) for which we have no ground truth for learning (no possible capture of these trajectories for model learning);
(ii) taking into account the third dimension (relative to the height of the pen), which impacts reconstruction performance;
(iii) switching to data acquired on various media, notably paper (for which we have no ground truth and whose signals are noisier due to friction on the paper).
Consequently, for the next stage of the project, we plan to implement a hybrid cost function to better guide the network in the joint learning on touching and hovering trajectories, as well as the integration of data acquired on different inclined planes to simulate trajectories in 3D space.
In addition, we intend to implement domain adaptation techniques to switch from data acquired on a tablet to data acquired on paper. These same methods will also be tested for different types of user (adult or child), to take account of differences in handwriting dynamics.

[1] Wassim Swaileh, Florent Imbert, Yann Soullard, Romain Tavenard, Eric Anquetil. Online Handwriting Trajectory Reconstruction from Kinematic Sensors using Temporal Convolutional Network. International Journal on Document Analysis and Recognition IJDAR (2023), 2023. ?hal-04076399v2?
[2] Toward Deep neural network for pen trajectory reconstruction from kinematic sensors Florent Imbert, Eric Anquetil, Romain Tavenard, Yann Soullard, Wassim Swaileh Symposium International Francophone sur l’Ecrit et le Document (SIFED’2022), Oct 2022, Rennes, France
[3] Adaptation de domaine pour la reconstruction de la trajectoire du stylo à partir de capteurs cinématiques Florent Imbert, Eric Anquetil, Romain Tavenard, Yann Soullard Symposium International Francophone sur l’Ecrit et le Document (SIFED’2023), Juin 2023, Paris, France

The aim of this project is to develop an intelligent learning device for automatic writing, composed of existing components, which can be made available to as many students as possible. The basis is the "Kaligo" application from the French company Learn&Go, which until now has been dependent on expensive tablet computers with a dedicated writing device and requires writing on the capacitive touchscreen. Due to the high acquisition costs of various individual components, these learning aids have not been able to hold their own on the market to date. The fusion of the electronic "DigiPen" from the STABILO company and the Kaligo app is intended to make precisely this possible: A cost-effective learning aid for the development of automated handwriting from which as many children as possible can benefit. In addition, the Digipen not only enables the use of commercially available tablets, but also writing on normal paper, which is a great advantage for the children's writing deve-lopment.
The French research institute IRISA, and more specifically the Intuidoc team, aims to carry out research on AI algorithms to automatically generate the on-line handwritten plot from signals produ-ced by digital pen sensors, while the German institute ITIV, part of the KIT, is studying different con-cepts to integrate AI algorithms adapted to embedded hardware. In this way, the complexity of the overall system is distributed over both software and hardware, enabling fast and efficient AI execu-tion. The objective is to enable online reconstruction of the pen trace from the digipen sensor data. A practical test with a demonstration system should prove the success of the work at the end of the three-year project.