Support of Advanced Test cOverage Criteria for RObust and Secure Software – SATOCROSS

The consortium includes two academic partners: CEA List (CEA) and the Interdisciplinary Centre for Security, Reliability and Trust (SnT) of Univ. of Luxembourg (UL), and one industrial partner MERCE (Mitsubishi Electric). The previous collaboration of the partners has convincingly demonstrated the benefits of the approach and proposed initial solutions for test generation and optimization for a small class of criteria. This collaboration helped to build an ambitious scientific program and motivated this project.

The partners will design generic and efficient testing techniques and tools for all major testing services: test input generation, detecting polluting test objectives and test assessment. They will focus on a large class of criteria important for industrial practice. Particular attention will be paid to the continuous development paradigm. The project will feature these tools in an integrated open-source toolkit that will be evaluated on real-life industrial code provided by the industrial partner. Finally, the project will establish industrial application guidelines that will allow practitioners to benefit from the project results.

Target coverage criteria and test optimization objectives (report)
Strategies for exhaustive branch coverage (report)

The project's results will be scientific, technological and industrial. SATOCROSS aims at developing cutting-edge technology in the area of software testing. This will have the potential of impacting software development practice and, more generally, the software industry and society. The open-source tools, publications and reports produced by the project will improve test automation and provide a wide range of users with greatly improved means to develop and maintain reliable software. Notably, software testing and customized solutions for continuous integration contribute to the DevOps culture and provide major competitive advantages (such as reduced time-to-market, improved software quality and reliability). SATOCROSS targets the central and most ‘expensive’ part of continuous integration (software testing) and as such has the potential to directly impact software development companies.

Thibault Martin, Nikolai Kosmatov, Virgile Prevosto, Matthieu Lemerre. Detection of Polluting Test Objectives for Dataflow Criteria.
In Proc.16th International Conference on Integrated Formal Methods (iFM) 16-20 November, 2020, Lugano, Switzerland, Lecture Notes in Computer Science, Springer, In press. ?cea-02974228?

Facundo Molina, Renzo Degiovanni, Pablo Ponzio, Germán Regis, Nazareno Aguirre, Marcelo F. Frias
Training binary classifiers as data structure invariants.
ICSE 2019: 759-770

Facundo Molina, César Cornejo, Renzo Degiovanni, Germán Regis, Pablo F. Castro, Nazareno Aguirre, Marcelo F. Frias
An evolutionary approach to translating operational specifications into declarative specifications.
Sci. Comput. Program. 181: 47-63 (2019)

Virgile Robles, Nikolai Kosmatov, Virgile Prévosto, Louis Rilling, Pascale Le Gall.
Tame your annotations with MetAcsl: Specifying, Testing and Proving High-Level Properties.
International Conference on Tests and Proofs (TAP), Oct 2019, Porto, Portugal. ?10.1007/978-3-030-31157-5_11?. ?cea-02301892?

Software testing practices have experienced rapid changes in the recent years. This can be attributed to the wide adoption of agile methodologies (such as continuous integration and test-driven development), which put testing at the center of software development. Testing all the possible program inputs is practically intractable and thus the software testing community has defined coverage criteria (a.k.a. adequacy criteria or testing criteria). These criteria are essential to an automated white-box testing process, as they are used by testing tools to guide the selection of new test cases, decide when testing should stop and assess the quality of a test suite. Dealing with advanced test criteria (such as boundary testing, MCDC, dafa-flow based criteria, etc.) is particularly important in safety critical applications. For instance, in regulated domains such as aeronautics, advanced coverage criteria are strict normative requirements that the tester must satisfy before delivering the software. In other domains, coverage criteria are recognized as a good practice for testing, and a key ingredient of test-driven development.

The existing testing tools support a limited number of test criteria in a hard-coded, non-generic manner. Therefore, in the current industrial practice, various tools are required to support different criteria. Moreover, in industry, in many cases test inputs are produced manually following informal criteria, based on the experience of test engineers in a particular domain. In this case such criteria are neither formalized nor supported by automated tools.

The SATOCROSS project (Support of Advanced Test cOverage Criteria for RObust and Secure Software) aims at improving software testing practice by bridging the gap between the potentialities offered by the huge body of academic work on (code-)coverage criteria on one side, and their limited tool support and use in the industry on the other side. The consortium includes two academic partners: CEA List (CEA) and the Interdisciplinary Centre for Security, Reliability and Trust (SnT) of Univ. of Luxembourg (UL), and one industrial partner MERCE (Mitsubishi Electric). The previous collaboration of the partners has convincingly demonstrated the benefits of the approach and proposed initial solutions for test generation and optimization for a small class of criteria. This collaboration helped to build an ambitious scientific program and motivated the submission of this proposal.

The partners will design generic and efficient testing techniques and tools for all major testing services: test input generation, detecting polluting test objectives and test assessment. They will focus on a large class of criteria important for the industrial practice. A particular attention will be paid to the continuous development paradigm. The project will feature these tools in an integrated open-source toolkit that will be evaluated on real-life industrial code provided by the industrial partner. Finally, the project will establish industrial application guidelines that will allow practitioners to benefit from the project results.

The major results of the project include scientific, technological, and industrial benefits. SATOCROSS aims at developing cutting edge technology in the area of software testing. This will have the potential of impacting the software development practice and more generally the software industry and society. The open-source tools, publications and reports produced by the project will improve test automation and will provide a wide range of users (software industry and practitioners) with greatly improved means to develop and maintain reliable software.