Hybrid AI towards Large Scale Data and Behavior Modeling for Automated Driving – HAIBrid

The project is built on a 3 step process: gather relevant traffic data with context, process it to create a large set of trajectories with semantics, and model interactions of road users. Each step brings innovation, but the complete chain brings the full value of the project.

Armines and KIT planned and designed data recording sessions and customized recording equipment, combining several Lidar sensors and cameras. An intersection in Karlsruhe, Germany, was chosen as the recording site due to the high number of interactions. Three permanent cameras were placed on a nearby high-rise building, offering a wide view of the entire intersection. For one day's recording, several Lidar sensors and cameras were placed on the junction's infrastructure. All the partners took part in the data recording, using their measurement vehicles to cross the intersection several times while being recorded on the infrastructure sensors. The result was 6 hours of precise data and 500 hours of recorded traffic data. As it was not necessary to switch off the permanent cameras, we continue to record data and have already exceeded 500 hours.

Three-dimensional and high-definition maps were created by FZI and SafeAD, the new partners who took over the work packages after Atlatec left the project consortium following its takeover by Bosch of Germany, which did not allow participation in the project. They recorded data to improve their AI-based mapping methods as planned.

Several publications have been accepted and others are being submitted. A seminar has been organized and patents are pending.

We propose to improve behavior prediction for autonomous driving by extracting trajectories and learning their interactions unsupervised from large scale data. We will leverage the experience and infrastructure of the consortium to collect a large body of traffic data from traffic cameras, drones and on-board vehicle sensors. Labeling vehicle trajectories and classifying interactions is hard due to their high variance in time and progression. Meanwhile, unsupervised methods push performance on image classification, segmentation and tracking, slowly approaching their supervised counterparts. Therefore, we will develop efficient methods for trajectory extraction and interaction clustering to take advantage of this recent progress but also generate 3D maps automatically. The static environment and the traffic rules, commonly encoded in semantic maps, are crucial not only to motion planning for automated vehicles but also to understand the behavior of other traffic participants. We will contribute collected data, developed methods and extracted knowledge about traffic participant trajectories and interactions to support the autonomous driving community and advance Franco-German AI competencies. Thereby the project will not only contribute novel multi-sensor recordings of scenes but methods and tools to support data extraction and labeling for future data recordings.