openebs-archive / cstor-csi

cStor CSI Driver

OpenEBS cStor CSI Driver

CSI driver implementation for OpenEBS cStor storage engine.

Project Status: Beta

The current implementation supports the following for cStor Volumes:

1. Provisioning and De-provisioning with ext4,xfs filesystems
2. Snapshots and clones
3. Volume Expansion
4. Volume Metrics

Setup OpenEBS cStor CSI Driver

OpenEBS cStor CSI driver components can be installed by using the helm chart or operator yaml. Refer to cstor-operators for detailed usage instructions.

How does it work?

OpenEBS cStor CSI driver comprises of 2 components:

• A controller component launched as a StatefulSet, implementing the CSI controller services. The Control Plane services are responsible for creating/deleting the required OpenEBS Volume.
• A node component that runs as a DaemonSet,implementing the CSI node services. The node component is responsible for performing the iSCSI connection management and connecting to the OpenEBS Volume.

The following steps indicate the PV provisioning workflow as it passes through various components.

1. Create PVC with Storage Class referring to OpenEBS cStor CSI Driver.

2. Kubernetes will pass the PV creation request to the OpenEBS CSI Controller service via CreateVolume(), as this controller registered with Kubernetes for receiving any requests related to cstor.csi.openebs.io

3. OpenEBS CSI Controller will create a custom resource called CStorVolumeConfig(CVC) and returns the details of the newly created object back to Kubernetes. The CVCs will be monitored by the cvc-operator. The cvc-operator watches the CVC resources and creates the respected volume specific resources like CStorVolume(CV), Target deployement, CStorVolumeReplicas and K8s service.

   Once the cStor Volume is created, the CVC is updated with the reference to the cStor Volume and change the status on CVC to bound.

4. Node Component which was waiting on the CVC status to be Bound will proceed to connect to the cStor volume.

Note: While the asynchronous handling of the Volume provisioning is in progress, the application pod may throw some errors like:

• Waiting for CVC to be bound: Implies volume components are still being created
• Volume is not ready: Replicas yet to connect to controller: Implies volume components are already created but yet to interact with each other.

On successful completion of the above steps the application pod can be seen in running state.

Contributing

OpenEBS welcomes your feedback and contributions in any form possible.

• Join OpenEBS community on Kubernetes Slack
  • Already signed up? Head to our discussions at #openebs
• Want to raise an issue or help with fixes and features?
  • See open issues
  • See contributing guide
  • See Project Roadmap
  • Want to join our contributor community meetings, check this out.
• Join our OpenEBS CNCF Mailing lists
  • For OpenEBS project updates, subscribe to OpenEBS Announcements
  • For interacting with other OpenEBS users, subscribe to OpenEBS Users

License

cstor-csi's Issues

Enhance csi Integration tests with new features

Describe the problem/challenge you have
[A description of the current limitation/problem/challenge that you are experiencing.]

Describe the solution you'd like
[A clear and concise description of what you want to happen.]

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• OpenEBS version (use kubectl get po -n openebs --show-labels):
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• kernel (e.g: uname -a):
• others:

Update target pod node selector

Describe the problem/challenge you have
I want to cluster the node selector to a specific cluster/zone because there is where the majority of the disk are running in.

Describe the solution you'd like
Let us update the CStor volume policy, and update the configuration dynamically according to the new settings.

Environment:

• OpenEBS version (use kubectl get po -n openebs --show-labels): 2.4.0
• Kubernetes version (use kubectl version): 1.19.4
• Cloud provider or hardware configuration: Custom built k3s/mixed cloud

Flaky CI e2e-test

CI checks seem to be failing consistently even for released versions.

No resource kind of CStorPoolCluster?

For the resizer feature,I want to create the CStorPoolCluster.But when create the resource,I get this error:

error: unable to recognize "cluster-pool.yaml": no matches for kind "CStorPoolCluster" in version "openebs.io/v1alpha1"

The cspc file is :

apiVersion: openebs.io/v1alpha1
kind: CStorPoolCluster
metadata:
  name: cstor
  namespace: openebs
  annotations:
    cas.openebs.io/config: |
      - name: Tolerations
        value: |-
          t1:
            effect: NoSchedule
            key: "node-role.kubernetes.io/master"
            operator: Exists
      - name: PoolResourceRequests
        value: |-
            memory: 2Gi
      - name: PoolResourceLimits
        value: |-
            memory: 4Gi
spec:
  pools:
    raidGroups:
    - type: "stripe"
      isWriteCache: false
      isSpare: false
      isReadCache: false
      blockDevices:
      - blockDeviceName: "blockdevice-263ec52bac19b2e65e663bff02224229"
      - blockDeviceName: "blockdevice-75ba6917c814053bc3c3b202cd8dbb10"
      - blockDeviceName: "blockdevice-8a19b43a88d8f294e3ee934367f6630d"
    poolConfig:
      cacheFile: ""
      defaultRaidGroupType: "stripe"
      overProvisioning: false
      compression: "off

Wirte this file by learn from https://raw.githubusercontent.com/openebs/csi/master/deploy/cspc.yaml

In addition， the openebs-operator version is 1.2,and the kubernetes version is 1.15.3.
If the openebs still not support CStorPoolCluster yet?

error generating accessibility requirements

Describe the bug: A clear and concise description of what the bug is.
Warning ProvisioningFailed 5m43s cstor.csi.openebs.io_openebs-cstor-csi-controller-0_7a41da72-a845-4d02-a602-703808f4d2be failed to provision volume with StorageClass "cstor-mirror-sc": error generating accessibility requirements: topologyKeys [] were not found on any nodes

Expected behaviour: A concise description of what you expected to happen
Volume created.

Steps to reproduce the bug:
A k8s cluster has kubevirt and cstor-csi. create pvc will produce the bug:
Warning ProvisioningFailed 5m43s cstor.csi.openebs.io_openebs-cstor-csi-controller-0_7a41da72-a845-4d02-a602-703808f4d2be failed to provision volume with StorageClass "cstor-mirror-sc": error generating accessibility requirements: topologyKeys [beta.kubernetes.io/arch beta.kubernetes.io/os cpu-feature.node.kubevirt.io/3dnowprefetch cpu-feature.node.kubevirt.io/abm cpu-feature.node.kubevirt.io/adx cpu-feature.node.kubevirt.io/aes cpu-feature.node.kubevirt.io/arat cpu-feature.node.kubevirt.io/arch-capabilities cpu-feature.node.kubevirt.io/avx cpu-feature.node.kubevirt.io/avx2 cpu-feature.node.kubevirt.io/bmi1 cpu-feature.node.kubevirt.io/bmi2 cpu-feature.node.kubevirt.io/clflushopt cpu-feature.node.kubevirt.io/clzero cpu-feature.node.kubevirt.io/cmp_legacy cpu-feature.node.kubevirt.io/cr8legacy cpu-feature.node.kubevirt.io/f16c cpu-feature.node.kubevirt.io/fma cpu-feature.node.kubevirt.io/fsgsbase cpu-feature.node.kubevirt.io/fxsr_opt cpu-feature.node.kubevirt.io/hypervisor cpu-feature.node.kubevirt.io/ibpb cpu-feature.node.kubevirt.io/invtsc cpu-feature.node.kubevirt.io/mds-no cpu-feature.node.kubevirt.io/misalignsse cpu-feature.node.kubevirt.io/mmxext cpu-feature.node.kubevirt.io/movbe cpu-feature.node.kubevirt.io/osvw cpu-feature.node.kubevirt.io/pclmuldq cpu-feature.node.kubevirt.io/pdpe1gb cpu-feature.node.kubevirt.io/perfctr_core cpu-feature.node.kubevirt.io/popcnt cpu-feature.node.kubevirt.io/pschange-mc-no cpu-feature.node.kubevirt.io/rdctl-no cpu-feature.node.kubevirt.io/rdrand cpu-feature.node.kubevirt.io/rdseed cpu-feature.node.kubevirt.io/rdtscp cpu-feature.node.kubevirt.io/sha-ni cpu-feature.node.kubevirt.io/skip-l1dfl-vmentry cpu-feature.node.kubevirt.io/smap cpu-feature.node.kubevirt.io/smep cpu-feature.node.kubevirt.io/sse4.2 cpu-feature.node.kubevirt.io/sse4a cpu-feature.node.kubevirt.io/tsc-deadline cpu-feature.node.kubevirt.io/tsc_adjust cpu-feature.node.kubevirt.io/virt-ssbd cpu-feature.node.kubevirt.io/vme cpu-feature.node.kubevirt.io/x2apic cpu-feature.node.kubevirt.io/xgetbv1 cpu-feature.node.kubevirt.io/xsave cpu-feature.node.kubevirt.io/xsavec cpu-feature.node.kubevirt.io/xsaveerptr cpu-feature.node.kubevirt.io/xsaveopt cpu-feature.node.kubevirt.io/xsaves cpu-model.node.kubevirt.io/Dhyana cpu-model.node.kubevirt.io/EPYC cpu-model.node.kubevirt.io/EPYC-IBPB cpu-model.node.kubevirt.io/Nehalem cpu-model.node.kubevirt.io/Opteron_G1 cpu-model.node.kubevirt.io/Opteron_G2 cpu-model.node.kubevirt.io/Opteron_G3 cpu-model.node.kubevirt.io/Penryn cpu-model.node.kubevirt.io/SandyBridge cpu-model.node.kubevirt.io/Westmere cpu-timer.node.kubevirt.io/tsc-frequency cpu-timer.node.kubevirt.io/tsc-scalable cpu-vendor.node.kubevirt.io/Hygon cpumanager host-model-cpu.node.kubevirt.io/Dhyana host-model-required-features.node.kubevirt.io/aes host-model-required-features.node.kubevirt.io/arch-capabilities host-model-required-features.node.kubevirt.io/clzero host-model-required-features.node.kubevirt.io/cmp_legacy host-model-required-features.node.kubevirt.io/hypervisor host-model-required-features.node.kubevirt.io/invtsc host-model-required-features.node.kubevirt.io/mds-no host-model-required-features.node.kubevirt.io/pclmuldq host-model-required-features.node.kubevirt.io/perfctr_core host-model-required-features.node.kubevirt.io/pschange-mc-no host-model-required-features.node.kubevirt.io/rdctl-no host-model-required-features.node.kubevirt.io/sha-ni host-model-required-features.node.kubevirt.io/skip-l1dfl-vmentry host-model-required-features.node.kubevirt.io/tsc-deadline host-model-required-features.node.kubevirt.io/tsc_adjust host-model-required-features.node.kubevirt.io/virt-ssbd host-model-required-features.node.kubevirt.io/x2apic host-model-required-features.node.kubevirt.io/xsaveerptr host-model-required-features.node.kubevirt.io/xsaves hyperv.node.kubevirt.io/base hyperv.node.kubevirt.io/frequencies hyperv.node.kubevirt.io/ipi hyperv.node.kubevirt.io/reenlightenment hyperv.node.kubevirt.io/reset hyperv.node.kubevirt.io/runtime hyperv.node.kubevirt.io/synic hyperv.node.kubevirt.io/synic2 hyperv.node.kubevirt.io/synictimer hyperv.node.kubevirt.io/time hyperv.node.kubevirt.io/tlbflush hyperv.node.kubevirt.io/vpindex kubernetes.io/arch kubernetes.io/hostname kubernetes.io/os kubevirt.io/schedulable meta.ecf.io/dedicated.customer meta.ecf.io/serverless node.cabinet openebs.io/nodeid openebs.io/nodename public-net.provider-network.kubernetes.io/interface public-net.provider-network.kubernetes.io/mtu public-net.provider-network.kubernetes.io/ready scheduling.node.kubevirt.io/tsc-frequency-1996194000 scheduling.node.kubevirt.io/tsc-frequency-1996213000 topology.cstor.openebs.io/nodeName trunk-net.provider-network.kubernetes.io/interface trunk-net.provider-network.kubernetes.io/mtu trunk-net.provider-network.kubernetes.io/ready vnf.ecf.io/schedulable vrouter.ecf.io/schedulable] were not found on any nodes

The output of the following commands will help us better understand what's going on:
kubectl get csinode 192.168.211.66 -oyaml

 - name: cstor.csi.openebs.io
    nodeID: 192.168.211.66
    topologyKeys:
    - beta.kubernetes.io/arch
    - beta.kubernetes.io/os
    - cpu-feature.node.kubevirt.io/3dnowprefetch
    - cpu-feature.node.kubevirt.io/abm
    - cpu-feature.node.kubevirt.io/adx
    - cpu-feature.node.kubevirt.io/aes
    - cpu-feature.node.kubevirt.io/amd-ssbd
    - cpu-feature.node.kubevirt.io/amd-stibp
    - cpu-feature.node.kubevirt.io/arat
    - cpu-feature.node.kubevirt.io/arch-capabilities
    - cpu-feature.node.kubevirt.io/avx
    - cpu-feature.node.kubevirt.io/avx2
    - cpu-feature.node.kubevirt.io/avx512bw
    - cpu-feature.node.kubevirt.io/avx512cd
    - cpu-feature.node.kubevirt.io/avx512dq
    - cpu-feature.node.kubevirt.io/avx512f
    - cpu-feature.node.kubevirt.io/avx512vl
    - cpu-feature.node.kubevirt.io/avx512vnni
    - cpu-feature.node.kubevirt.io/bmi1
    - cpu-feature.node.kubevirt.io/bmi2
    - cpu-feature.node.kubevirt.io/clflushopt
    - cpu-feature.node.kubevirt.io/clwb
    - cpu-feature.node.kubevirt.io/erms
    - cpu-feature.node.kubevirt.io/f16c
    - cpu-feature.node.kubevirt.io/fma
    - cpu-feature.node.kubevirt.io/fsgsbase
    - cpu-feature.node.kubevirt.io/hypervisor
    - cpu-feature.node.kubevirt.io/ibpb
    - cpu-feature.node.kubevirt.io/ibrs
    - cpu-feature.node.kubevirt.io/ibrs-all
    - cpu-feature.node.kubevirt.io/invpcid
    - cpu-feature.node.kubevirt.io/invtsc
    - cpu-feature.node.kubevirt.io/md-clear
    - cpu-feature.node.kubevirt.io/mds-no
    - cpu-feature.node.kubevirt.io/movbe
    - cpu-feature.node.kubevirt.io/mpx
    - cpu-feature.node.kubevirt.io/pcid
    - cpu-feature.node.kubevirt.io/pclmuldq
    - cpu-feature.node.kubevirt.io/pdpe1gb
    - cpu-feature.node.kubevirt.io/pku
    - cpu-feature.node.kubevirt.io/popcnt
    - cpu-feature.node.kubevirt.io/pschange-mc-no
    - cpu-feature.node.kubevirt.io/rdctl-no
    - cpu-feature.node.kubevirt.io/rdrand
    - cpu-feature.node.kubevirt.io/rdseed
    - cpu-feature.node.kubevirt.io/rdtscp
    - cpu-feature.node.kubevirt.io/skip-l1dfl-vmentry
    - cpu-feature.node.kubevirt.io/smap
    - cpu-feature.node.kubevirt.io/smep
    - cpu-feature.node.kubevirt.io/spec-ctrl
    - cpu-feature.node.kubevirt.io/ss
    - cpu-feature.node.kubevirt.io/ssbd
    - cpu-feature.node.kubevirt.io/sse4.2
    - cpu-feature.node.kubevirt.io/stibp
    - cpu-feature.node.kubevirt.io/tsc-deadline
    - cpu-feature.node.kubevirt.io/tsc_adjust
    - cpu-feature.node.kubevirt.io/umip
    - cpu-feature.node.kubevirt.io/vme
    - cpu-feature.node.kubevirt.io/vmx
    - cpu-feature.node.kubevirt.io/x2apic
    - cpu-feature.node.kubevirt.io/xgetbv1
    - cpu-feature.node.kubevirt.io/xsave
    - cpu-feature.node.kubevirt.io/xsavec
    - cpu-feature.node.kubevirt.io/xsaveopt
    - cpu-feature.node.kubevirt.io/xsaves
    - cpu-model.node.kubevirt.io/Broadwell-noTSX
    - cpu-model.node.kubevirt.io/Broadwell-noTSX-IBRS
    - cpu-model.node.kubevirt.io/Cascadelake-Server-noTSX
    - cpu-model.node.kubevirt.io/Haswell-noTSX
    - cpu-model.node.kubevirt.io/Haswell-noTSX-IBRS
    - cpu-model.node.kubevirt.io/IvyBridge
    - cpu-model.node.kubevirt.io/IvyBridge-IBRS
    - cpu-model.node.kubevirt.io/Nehalem
    - cpu-model.node.kubevirt.io/Nehalem-IBRS
    - cpu-model.node.kubevirt.io/Opteron_G1
    - cpu-model.node.kubevirt.io/Opteron_G2
    - cpu-model.node.kubevirt.io/Penryn
    - cpu-model.node.kubevirt.io/SandyBridge
    - cpu-model.node.kubevirt.io/SandyBridge-IBRS
    - cpu-model.node.kubevirt.io/Skylake-Client-noTSX-IBRS
    - cpu-model.node.kubevirt.io/Skylake-Server-noTSX-IBRS
    - cpu-model.node.kubevirt.io/Westmere
    - cpu-model.node.kubevirt.io/Westmere-IBRS
    - cpu-timer.node.kubevirt.io/tsc-frequency
    - cpu-timer.node.kubevirt.io/tsc-scalable
    - cpu-vendor.node.kubevirt.io/Intel
    - cpumanager
    - host-model-cpu.node.kubevirt.io/Cascadelake-Server
    - host-model-required-features.node.kubevirt.io/amd-ssbd
    - host-model-required-features.node.kubevirt.io/amd-stibp
    - host-model-required-features.node.kubevirt.io/arch-capabilities
    - host-model-required-features.node.kubevirt.io/hypervisor
    - host-model-required-features.node.kubevirt.io/ibpb
    - host-model-required-features.node.kubevirt.io/ibrs
    - host-model-required-features.node.kubevirt.io/ibrs-all
    - host-model-required-features.node.kubevirt.io/invtsc
    - host-model-required-features.node.kubevirt.io/md-clear
    - host-model-required-features.node.kubevirt.io/mds-no
    - host-model-required-features.node.kubevirt.io/pku
    - host-model-required-features.node.kubevirt.io/pschange-mc-no
    - host-model-required-features.node.kubevirt.io/rdctl-no
    - host-model-required-features.node.kubevirt.io/skip-l1dfl-vmentry
    - host-model-required-features.node.kubevirt.io/ss
    - host-model-required-features.node.kubevirt.io/stibp
    - host-model-required-features.node.kubevirt.io/tsc_adjust
    - host-model-required-features.node.kubevirt.io/umip
    - host-model-required-features.node.kubevirt.io/vmx
    - host-model-required-features.node.kubevirt.io/xsaves
    - hyperv.node.kubevirt.io/base
    - hyperv.node.kubevirt.io/frequencies
    - hyperv.node.kubevirt.io/ipi
    - hyperv.node.kubevirt.io/reenlightenment
    - hyperv.node.kubevirt.io/reset
    - hyperv.node.kubevirt.io/runtime
    - hyperv.node.kubevirt.io/synic
    - hyperv.node.kubevirt.io/synic2
    - hyperv.node.kubevirt.io/synictimer
    - hyperv.node.kubevirt.io/time
    - hyperv.node.kubevirt.io/tlbflush
    - hyperv.node.kubevirt.io/vpindex
    - kube-ovn/role
    - kubernetes.io/arch
    - kubernetes.io/hostname
    - kubernetes.io/os
    - kubevirt.io/schedulable
    - meta.ecf.io/dedicated.customer
    - meta.ecf.io/serverless
    - node-role.kubernetes.io/master
    - node.cabinet
    - openebs.io/nodeid
    - openebs.io/nodename
    - public-net.provider-network.kubernetes.io/interface
    - public-net.provider-network.kubernetes.io/mtu
    - public-net.provider-network.kubernetes.io/ready
    - scheduling.node.kubevirt.io/tsc-frequency-1996194000
    - scheduling.node.kubevirt.io/tsc-frequency-2294577000
    - topology.cstor.openebs.io/nodeName
    - trunk-net.provider-network.kubernetes.io/interface
    - trunk-net.provider-network.kubernetes.io/mtu
    - trunk-net.provider-network.kubernetes.io/ready
    - vnf.ecf.io/schedulable
    - vrouter.ecf.io/schedulable

Anything else we need to know?:
I found csiNode TopologKeys:
- scheduling.node.kubevirt.io/tsc-frequency-1996194000
- scheduling.node.kubevirt.io/tsc-frequency-2294577000
not match the pvc request TopologKeys.

Environment details:

• OpenEBS version (use kubectl get po -n openebs --show-labels):
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• kernel (e.g: uname -a):
• others: