Safety check throughout testing procedure about sieve

[marshtompsxd]

@kosyd Thank you so much for describing the example! And yes checking safety property is a very important feature that we are implementing.

To check safety properties, Sieve automatically instruments the apisever and records the execution history of the whole k8s cluster by intercepting various (Creation/Deletion/Update) events for all kinds of resources (including both custom resources and k8s core resources). In that way, Sieve can capture every single change to each resource and Sieve can check the resource state for any object at any point by scanning the recorded history.

For now, this feature is only for internal use and we are actively developing the APIs (and documentation) for external users to customize their oracles in a declarative way and check for various safety properties. In the particular example you described, the user can write a declarative oracle asserting that "the pod ratio between two specific deployments should always be no larger than X%". After that Sieve will traverse the history and check the property at every point in the history where the deployment has changed. If the ratio is incorrect at any point, Sieve will capture it.

We will definitely let you know when the feature is ready for use and we will also prepare some examples on how to customize the declarative oracles.

[marshtompsxd]

Hello @kosyd
Sorry for the late response. Sieve currently implements customized safety checker that allows users to specify any safety property that checks any resources throughout the testing procedure. You can find examples here: https://github.com/sieve-project/sieve/blob/main/sieve_oracle/customized_safety_checker.py

To implement the safety checker, all a user needs to do is to specify the checker function and the resource to check.

Is it possible for you to point us to some concrete controller example that you want to check the safety property? So that we can know whether this is the exact feature you need. It would also be great if you could try Sieve to test some of your controllers and let us know any feedback.