Deploy on Production environments about csi-lvm

[majst01]

Hi,

1. Docker images are here:
   https://hub.docker.com/r/metalstack/csi-lvm-controller
   https://hub.docker.com/r/metalstack/csi-lvm-provisioner
2. This is by design, after reboot the reviver pod will remount all PVs as before. You cant rely on stuff installed/configured on the machine on a kubernetes node because this might always be gone. kubelet reconcilation must take care.

Please give more details like:

• k8s version
• csi-lvm version
• your manifests you deployed csi-lvm

[rdrgporto]

Hi @majst01,

• k8s version: 1.15.4
• csi-lvm: 0.6.0

I have seen that 0.6.1 is not available on Docker Hub. I have modified manifests in order to use 0.6.0 version.

As I said above, I have attached a new disk (/dev/sdb) and I had to create a Physical Volume and Volume Group:

sudo pvcreate /dev/sdb1
sudo vgcreate csi-lvm /dev/sdb1

It is neccesary to do that ?

I have modified controller deployment manifest in order to use /dev/sdb1:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: csi-lvm-controller
  namespace: csi-lvm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: csi-lvm-controller
  template:
    metadata:
      labels:
        app: csi-lvm-controller
    spec:
      serviceAccountName: csi-lvm-controller
      containers:
      - name: csi-lvm-controller
        image: metalstack/csi-lvm-controller:v0.6.0
        imagePullPolicy: IfNotPresent
        command:
        - /csi-lvm-controller
        args:
        - start
        env:
        - name: CSI_LVM_PROVISIONER_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: CSI_LVM_PULL_POLICY
          value: "IfNotPresent"
        - name: CSI_LVM_PROVISIONER_IMAGE
          value: "metalstack/csi-lvm-provisioner:v0.6.0"
        - name: CSI_LVM_DEVICE_PATTERN
          value: "/dev/sdb1"

Finally, I have deployed a Statefulset example:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: local-test
spec:
  serviceName: "local-service"
  replicas: 2
  selector:
    matchLabels:
      app: local-test-new
  template:
    metadata:
      labels:
        app: local-test-new
    spec:
      containers:
      - name: test-container
        image: k8s.gcr.io/busybox
        command:
        - "/bin/sh"
        args:
        - "-c"
        - "count=0; count_file=\"/usr/test-pod/count\"; test_file=\"/usr/test-pod/test_file\"; if [ -e $count_file ]; then count=$(cat $count_file); fi; echo $((count+1)) > $count_file; while [ 1 ]; do date >> $test_file; echo \"This is $MY_POD_NAME, count=$(cat $count_file)\" >> $test_file; sleep 10; done"
        volumeMounts:
        - name: local-vol
          mountPath: /usr/test-pod
        env:
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
      securityContext:
        fsGroup: 1234
  volumeClaimTemplates:
  - metadata:
      name: local-vol
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "csi-lvm"
      resources:
        requests:
          storage: 1Gi

It created pv, pvc and pods correctly. After rebooting, csi-lvm-reviver created pvc in /tmp/csi-lvm again, so that, it worked.

Perphaps, the last time, I did something wrong or I did not have enough patience.

Thanks,

Regards

[majst01]

No you cant create the csi-lvm vg by hand this will be done by csi-lvm provisioner at the first time if it is not present.

v0.6.1 is not tagged yet, but already put into the manifest,sorry for that.

[rdrgporto]

Hi @majst01,

I have done the following steps:

• Attach a new disk (/dev/sdb)
• Modify Controller manifest
• Deploy Controller and Reviver

Reviver logs:

I0327 10:39:35.409800       1 revivelvs.go:68] starting reviver
I0327 10:39:35.456983       1 createlv.go:217] unable to list existing volumegroups:expected 5 components, got 1
I0327 10:39:35.457099       1 revivelvs.go:79] volumegroup: csi-lvm not found
I0327 10:39:35.577278       1 createlv.go:217] unable to list existing volumegroups:expected 5 components, got 1
I0327 10:39:35.577433       1 revivelvs.go:84] volumegroup: csi-lvm not found
I0327 10:39:35.620551       1 revivelvs.go:57] vgs output:
I0327 10:39:35.667161       1 revivelvs.go:63] lvs output:

Controller logs:

I0327 10:39:35.680418       1 main.go:169] Provisioner started
I0327 10:39:35.683145       1 leaderelection.go:242] attempting to acquire leader lease  csi-lvm/metal-stack.io-csi-lvm...
I0327 10:39:35.699498       1 leaderelection.go:252] successfully acquired lease csi-lvm/metal-stack.io-csi-lvm
I0327 10:39:35.700894       1 controller.go:780] Starting provisioner controller metal-stack.io/csi-lvm_csi-lvm-controller-54445fbdb8-jw55w_1f777e1e-c737-4e9b-ba8c-3ad7d03ceaa4!
I0327 10:39:35.700979       1 event.go:281] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"csi-lvm", Name:"metal-stack.io-csi-lvm", UID:"f0037d7a-d40b-42ba-a32d-40455fc24381", APIVersion:"v1", ResourceVersion:"11733", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' csi-lvm-controller-54445fbdb8-jw55w_1f777e1e-c737-4e9b-ba8c-3ad7d03ceaa4 became leader
I0327 10:39:35.801235       1 controller.go:829] Started provisioner controller metal-stack.io/csi-lvm_csi-lvm-controller-54445fbdb8-jw55w_1f777e1e-c737-4e9b-ba8c-3ad7d03ceaa4!

Thanks in advance,

Regards

[majst01]

Yes but once you create a pv, the volumegroup and then the lv will be created

[rdrgporto]

Hi @majst01,

Without creating a pv and vg before (as I said above), it does not works. If I create pv, vg, it works.

Regards

[majst01]

Wierd, we run integration tests which cover this topic and have a bunch of clusters on bare-metal without problems you describe

[rdrgporto]

Hi @majst01,

Thanks for all.

Regards