Java Unit Testing Competition and CPS Testing Tool Competition Results from SBST 2021
COSMOS project partners Zurich University of Applied Sciences and University of Sannio hosted 9th edition of the Java Unit Testing Tool Competition and 1st edition of the Cyber-Physical Systems Testing Tool Competition on 31 May 2021, as part of the International Workshop on Search-Based Software Testing (SBST 2021). The slides presented are available online. Key results from the competition were:
Java Unit Testing Competition: This year, five tools, Randoop, UtBot, Kex, Evosuite, and EvosuiteDSE, were executed on a benchmark with: (i) new classes under test, selected from three open-source software projects; and (ii) the set of classes from three projects considered in the eighth edition. We relied on an improved Docker infrastructure to execute the tools and the subsequent coverage and mutation analysis. Given the high number of participants, we considered only two time budgets for test case generation: thirty seconds and two minutes. The Winner of the Java Unit Testing Competition was Evosuite, followed in the ranking by Randoop, EvosuiteDSE, Utbot, and Kex.
Cyber-Physical Systems Testing Tool Competition: Five tools, Deeper, Frenetic, GABExplore, GABExploit, and Swat, competed on testing self-driving car software by generating simulation-based tests using our new testing infrastructure. We considered two experimental settings to study test generators' transitory and asymptotic behaviours and evaluated the tools' test generation effectiveness and the exposed failures' diversity. The Winner of the Cyber-Physical Systems Testing Tool Competition was Deeper, followed in the ranking by Frenetic, GABExploit, GABExplore, and Swat.
Both of these competitions were important for raising awareness of the state-of-art in testing capabilities within key applications domains being addressed by the COSMOS project.
