sujeshthekkepatt / absurd-ci

Design

1. Create the AbsurdCI CR

2. The controller will init the statuses

3. Controller will go through the steps

4. Controller will init a PVC

5. Every step is ran inside a POD where every commands ran inside an init container to preserve the order

   a. There will be additional init container which will prepare the working directory. It will clone the git repo. This will be the working directory of the containers.

   b. Then we will create additional init containers which will have the commands specified in the steps

   c. There will be a common container which may collect the logs and update the CR Statuses. This may be a new pod also which will collect everything. In that case the last pod will update the CR

   d. Based on the CR updations the controller will get triggered again and will launch the next step in a new pod.

   e. This process happens till all the steps are executed

6. We will use PVC to share data between multiple steps. Unless specified each steps will have same working directory. Make sure to use a storge type which support "ReadWriteMany" option. Else we need to delete pods that completed execution.

7. The pods may get deleted or maynot in the POC

8. The logs of execution can be aggragted to a file in the PVC and end of the steps we can use a pod to collect the evidence.

CR status

1. PodExecutionContext{}
   • TotalNumberOfSteps
   • TotalNumberOfTasks
   • CurrentStep
   • NextStep
   • TotalNumberOfTasksCompleted
   • TotalNUmberOfStepsCompleted
   • TotalExecutionTime
   • Namespace
2. StepPodInfo
   • StepName
   • PodName
   • IsPodCreated
   • ConatinerNames []{}
     • ContainerName
     • CommandStatus
     • CommandLog

absurd-ci's Issues

Use individual pods other than init containers

As part of #6 now the steps are running inside a container rather than a init containers. We can give multiple commands to a step. All the commands in a step will run parallel (default container execution model).

Set proper CR statuses

We have setup topological sort and thus now we can schedule pods. We need to know effectively the status of the pod before running next step. If the step fails we need to exit the pipeline.

Allow scripts to be executed with the container

Currently each step runs a single command. We need the provision to run scripts as part of the step.

Setup pod scheduling via topological sorting

Currently we are running jobs via init containers which ensures ordering of tasks but lacks flexibility on manipulation. We need to implement #5. The initial scheudling we can do via topological sorting and then we can further optimize from there. This is an initial thought.

We need to include some new fields into CRD which signifies the relationship between different tasks. Ideally we need to have a runAfter field points to a step name in the CR

Add support for mounting secrets in the task pod

User can specify a secret name and can specify how that secret needs to be attached with the pod. Possible ways are ,

• Env

• Mounting inside the workspace at a specified path
  This will be needed when we need to authenticate external systems/ private repositories etc from the pod

• Mount the user specified secret in the workspace or particular path in the pod specified by the user

• Mount the secret in the pod env values using envFrom

Controller unresponsive and needs restart

InitPVC throws error when PVC already exists

Currently InitPVC function throws error if already PVC exists