Members of the MATRIS Research Group participated in the 14th International Workshop on Combinatorial Testing (IWCT 2025) in Naples, Italy. IWCT is among the premier venues for research dedicated to combinatorial testing and was held co-located with the 18th ICST on Monday, March 31, 2025.
Andrea Bombarda (University of Bergamo) and Bernhard Garn (MATRIS) served as PC Co-Chairs together with General Chair Angelo Gargantini (University of Bergamo).
IWCT 2025 proudly featured a keynote given by Dr. Juraj Somorovsky, Professor at Paderborn University (Germany), titled “Lessons learned from systematic security testing of TLS”: Transport Layer Security (TLS) is one of the most important cryptographic protocols, securing a vast range of communications, including web traffic, email, chat, and VPNs. Given its widespread importance, TLS implementations have become prime targets for numerous famous attacks. To detect TLS attacks automatically, Juraj – together with his System Security Group at UPB – developed a flexible tool for TLS security evaluations called TLS-Attacker. He demonstrated how combinatorial testing integrated into TLS-Attacker leveraged a practical TLS test suite called TLS-Anvil that found new exploits, cryptographic problems, and interoperability issues.
Manuel Leithner presented a paper on the practical integration of Combinatorial (Security) Testing workflows in Continuous Integration and Deployment environments (joint work with Jovan Zivanovic, Reinhard Kugler and Dimitris Simos), continuing to support a shift of combinatorial testing methods towards practical applicability.
Bernhard Garn presented the paper titled “Combinatorial Methods for Enhancing the Resilience of Production Facilities”; joint work with Konstantin Gerner and Wolfgang Czerni (both Infraprotect), Rick Kuhn and Raghu Kacker (both NIST) and Klaus Kieseberg and Dimitris Simos. In their paper, they applied combinatorial methods to generate crisis scenarios for production facilities with the goal of strengthening their resilience by identifying weaknesses in operational aspects or crisis response plans, extending previous work from the domain of disaster research. This paper highlights the wide range of topics covered at IWCT 2025.
More applications of combinatorial testing were presented in the papers:
- “A Search-Based Benchmark Generator for Constrained Combinatorial Testing Models” by Paolo Arcaini (National Institute of Informatics), Andrea Bombarda (University of Bergamo), Angelo Gargantini (University of Bergamo)
- “Combinatorial Test Design Model Creation using Large Language Models” by Deborah Furman (IBM), Eitan Farchi (IBM Haifa Research Lab), Michael Gildein (IBM), Andrew Hicks (IBM), Ryan Rawlins (IBM)
- “Evaluating Large Language Model Robustness Using Combinatorial Testing” by Jaganmohan Chandrasekaran (Virginia Tech), Ankita Ramjibhai Patel (The University of Texas at Arlington), Erin Lanus (Virginia Tech), Laura Freeman (Virginia Tech)
- “Utilizing Ontologies for Combinatorial Testing” by Franz Wotawa (Graz University of Technology)
- “Towards Accessibility of Covering Arrays for Practitioners of Combinatorial Testing” by Ulrike Grömping (BHT – Berliner Hochschule für Technik)
- “Testing Tool for Combinatorial Transition Testing in Dynamically Adaptive Software Systems” by Pierre Martou (UCLouvain / ICTEAM), Benoît Duhoux (Université catholique de Louvain), Kim Mens (Université catholique de Louvain, ICTEAM), Axel Legay (Nexova)
- “Extended Abstract of Poster: STARS: Tree-based Classification and Testing of Feature Combinations in the Automated Robotic Domain” by Till Schallau (TU Dortmund University), Dominik Schmid (TU Dortmund University), Nick Pawlinorz (TU Dortmund University), Stefan Naujokat (TU Dortmund University), Falk Howar (TU Dortmund University)
- “Data Frequency Coverage Impact on AI Performance” by Erin Lanus (Virginia Tech), Brian Lee (Virginia Tech), Jaganmohan Chandrasekaran (Virginia Tech), Laura Freeman (Virginia Tech), M S Raunak (NIST), Raghu Kacker (NIST), Rick Kuhn (NIST)
- “A Combinatorial Approach to Reduce Machine Learning Dataset Size” by Megan Olsen (Loyola University Maryland), M S Raunak (NIST), Rick Kuhn (NIST), Fenrir Badorf (Loyola University Maryland), Hans van Lierop (Loyola University Maryland), Francis Durso (Johns Hopkins University)
- “Fairness Testing of Machine Learning Models Using Combinatorial Testing in Latent Space” by Arjun Dahal (University of Texas at Arlington), Sunny Shree (University of Texas at Arlington), Jeff Lei (University of Texas at Arlington), Raghu Kacker (NIST), Rick Kuhn (NIST)
Dimitris Simos, in his capacity as IWCT Steering Committee Chair, formulated the strategic direction of the workshop in cooperation with members of the Steering Committee, bolstered by great suggestions provided by the IWCT community in attendance.
On behalf of the Organizers of IWCT 2025, we would like to thank all authors of accepted papers, all presenters and all participants! Furthermore, we would like to thank the Organizing Committee of ICST 2025 for their support; with special appreciation to the ICSTW Co-Chairs Matteo Biagiola, Mitchell Olsthoorn and Francesca Lonetti, the Finance Co-Chairs Domenico Amalfitano and Vincenzo Riccio, the Registration Chairs Aurora Ramírez and Baharin Aliashrafi Jodat as well as the Local Arrangement Chair Alessandra De Benedictis!
