tthuem / samplingchallenge Goto Github PK
View Code? Open in Web Editor NEWA repository to plan upcoming versions of the SPLC challenge on product sampling
A repository to plan upcoming versions of the SPLC challenge on product sampling
When allowing the submission of sampling algorithms that do not ensure t-wise coverage, we need to be able to compute the percentage of t-wise interactions being covered.
This is also useful to check for algorithms with t-wise coverage in two dimensions:
After a user provided sampling algorithms and calculated a sample he should get an evaluation how well his sampling algorithm performed. To enable this functionality the following evaluation criteria need to be integrated into the TIRA system.
The scalability challenge is all about testing the boundaries of sampling procedures. To do so, we need to include large feature models into the TIRA system. For a first prototype feature models such as, Automotive01, Automotive02, and FinancialServices01 must be added to the system. Based on these data first performance tests of the system can be done. For instance, checking how many resources (CPU, RAM, etc.) is need in each virtual machine.
A core functionality of the challenge is that users can provide a sampling algorithm, which calculates samples for a given t-wise coverage. To test out this workflow an example algorithm, such as IncLing, is needed. The algorithm must executable as comand line application to be used in the TIRA system. Hence, this issue includes not only the integration of a sampling algorithm in TIRA, but also preparing the algorithm as a comand line application. In case of IncLing there is already a prototype implementation of a headless version (https://github.com/PettTo/Master_Thesis_Tools/tree/master/HeadlessIncling), which should be used as basis for further work.
State-of-the-art algorithms are typically oblivious to the evolution of the product line. They can still be applied to the evolution, but may be applied in a separate category that evaluates algorithms under feature-model evolution.
Furthermore, we may introduce a new input format for dedicated sampling algorithms that need several feature-model revisions at the same time.
While most sampling algorithms only consume the feature model as a propositional formula, there are a few algorithms that do also incorporate the feature hierarchy of the feature diagram. Not all benchmark models are useful in those cases (i.e., those being imported from DIMACS do not have a hierachy). Also, we might need a special input format or at least a separate category for comparison of outputs.
One essential part of automating the sampling challenge is to integrate feature models into the system. By starting with small feature models such as GPL, FeatureIDE, FameDB and EShop a test suite for validation of newly integrated functionality can be build.
While most sampling algorithms only accepted boolean features and constraints among those, there are also other algorithms that can cope with numerical features.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.