Automata Learning and Testing in AALpy
The tutorial will introduce automata learning and its relation to black-box testing with the help of our AALpy tool. This learning-based testing approach combines the active learning of finite-state models with model-based test-case generation. With AALpy we can learn (and test against) deterministic, non-deterministic, and stochastic models. After demonstrating the automated testing of Python classes, I will discuss more sophisticated applications, including protocol fuzzing (MQTT, Bluetooth, IPSec) and the mining of interpretable models from recurrent neural networks.
Property-based testing: new developments
In this pair of lectures I will introduce property-based testing, using the Haskell and Erlang instances as examples, and I will try to give a picture of how far the technique has spread today, and where and how it is best deployed. I will aim to talk about some of the new kinds of properties we have begun using in recent years, in particular those based on using dynamic logic to express liveness as well as safety properties of smart contracts.
Testing Cyber Physical Systems: Dynamic Modeling, Evolutionary Search and Machine Learning
The development of a cyber-physical system (CPS) largely relies on early functional and dynamic models of the system and its environment. These models are executable, exhibit time-varying behaviours and capture physical, communication and algorithmic aspects of CPS. Traditionally, CPS testing techniques have been rooted in logic-based and algorithmic verification. In this talk, we review alternative CPS testing techniques that leverage a combination of evolutionary search and machine learning. I will start with an introduction to search-based testing and then describe its applications for model-in-the-loop (MiL) testing — a common practice in the CPS industry. Using autonomous driving systems as a case study, I will discuss different heuristics to scale up search-based testing for handling complex CPS systems. The talk further outlines how, using interpretable machine learning, one can generate environment conditions that characterize system correctness or, alternatively, explain system failures.
Combinatorial Interaction Testing
The tutorial’s focus will be Combinatorial Interaction Testing (CIT), a mature area of software testing that benefited from search-based software engineering techniques (SBSE). CIT is a black-box testing method that has been extensively used in testing highly configurable software. In practice it’s usually infeasible to test all possible parameter combinations of such systems. Empirical studies have shown that CIT provides a way of reducing the test suite size whilst still maintaining high fault detection rates. SBSE techniques have been successfully applied to the CIT test suite generation problem which will be presented during the tutorial.
Towards Correct and Reliable Data-centric Systems
Data-centric systems are software that store, retrieve, manipulate, or process data. They are the backbone of our computing infrastructure and thus of unparalleled importance. The goal of this tutorial is to demonstrate how automated testing can find bugs in such systems, focusing on database engines. Specifically, the tutorial will present ways how to tackle test case generation as well as the test oracle problem in this context. Besides a theoretical component, the tutorial will involve a hands-on component, showing participants how to quickly prototype their ideas.
Common fuzzing techniques work by systematically mutating a set of given inputs, slowly covering more and more of the program code. But if your program has a complex input format, most of these mutations will be invalid, resulting in very few inputs reaching code beyond input processing. In this tutorial, we will explore techniques to specify input languages using grammars, generators, and constraint solvers, and leverage these language specifications to create powerful fuzzers for complex input formats. Includes live coding and exercises.
Recommended reads. fuzzingbook.org, notably the chapters “Fuzzing with Grammars”, “Fuzzing with Generators”, and “Fuzzing with Constraints”