tthuem / samplingchallenge Goto Github PK

A core functionality of the challenge is that users can provide a sampling algorithm, which calculates samples for a given t-wise coverage. To test out this workflow an example algorithm, such as IncLing, is needed. The algorithm must executable as comand line application to be used in the TIRA system. Hence, this issue includes not only the integration of a sampling algorithm in TIRA, but also preparing the algorithm as a comand line application. In case of IncLing there is already a prototype implementation of a headless version (https://github.com/PettTo/Master_Thesis_Tools/tree/master/HeadlessIncling), which should be used as basis for further work.

While most sampling algorithms only consume the feature model as a propositional formula, there are a few algorithms that do also incorporate the feature hierarchy of the feature diagram. Not all benchmark models are useful in those cases (i.e., those being imported from DIMACS do not have a hierachy). Also, we might need a special input format or at least a separate category for comparison of outputs.

One essential part of automating the sampling challenge is to integrate feature models into the system. By starting with small feature models such as GPL, FeatureIDE, FameDB and EShop a test suite for validation of newly integrated functionality can be build.

The scalability challenge is all about testing the boundaries of sampling procedures. To do so, we need to include large feature models into the TIRA system. For a first prototype feature models such as, Automotive01, Automotive02, and FinancialServices01 must be added to the system. Based on these data first performance tests of the system can be done. For instance, checking how many resources (CPU, RAM, etc.) is need in each virtual machine.

After a user provided sampling algorithms and calculated a sample he should get an evaluation how well his sampling algorithm performed. To enable this functionality the following evaluation criteria need to be integrated into the TIRA system.

• Make sure samples are saved persistently?
• Count the number of configurations contained in the sample
• Calculate how long a sampling algorithm needs to calculate a solution.
• Track memory consumption during solution calculation
• Calculate percentage of achieved t-wise coverage (see Issue#1)

When allowing the submission of sampling algorithms that do not ensure t-wise coverage, we need to be able to compute the percentage of t-wise interactions being covered.

This is also useful to check for algorithms with t-wise coverage in two dimensions:

1. We can identify faults in algorithms.
2. We can check the coverage for higher values of t for those algorithms too.

While most sampling algorithms only accepted boolean features and constraints among those, there are also other algorithms that can cope with numerical features.

State-of-the-art algorithms are typically oblivious to the evolution of the product line. They can still be applied to the evolution, but may be applied in a separate category that evaluates algorithms under feature-model evolution.

Furthermore, we may introduce a new input format for dedicated sampling algorithms that need several feature-model revisions at the same time.