kubernetes-sigs / nfs-ganesha-server-and-external-provisioner Goto Github PK
View Code? Open in Web Editor NEWNFS Ganesha Server and Volume Provisioner.
License: Apache License 2.0
NFS Ganesha Server and Volume Provisioner.
License: Apache License 2.0
Hi there,
I would like to know which perms you recommend about the directory that will be exported by the NFS Storage Class.
Many thanks
Volume resizing and splitting servers by subpaths has two problems in my usage:
One solution might be a different Provisioner that provisions NFS servers and the storage that back them according to parameters of the storage class, the PVC that triggered provisioning, or the pod that bound the triggering PVC (if volumeBindingMode: WaitForFIrstConsumer
)
Mode 1: If the user wants to be able to provision PVCs ahead of time.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-fast
labels:
storageClass: 'fast'
provisioner: cluster.local/multi-nfs-provisioner
parameters:
storageClass: 'fast'
resources:
requests:
cpu: 100/1Ti
memory: 250/1Ti
topology:
...
affinities:
...
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: Immediate
Mode2: User is ok with slower provisioning in exchange for better utilization, and keeping the server near its consumers.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-fast-dynamic
labels:
storageClass: 'fast'
parameters:
storageClass: 'fast'
resources:
requests:
cpu: 100m/1Ti
memory: 250m/1Ti
provisioner: cluster.local/multi-nfs-provisioner
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
When a new volume is requested with storage class nfs-fast
A nfs server backed by a dynamically provisioned volume of storageClass
with topology/affinity constraints either from parameters or the bound pod, and size from the requesting PVC.
Additionally resource requests could be specified in terms of the storage size requested, since for most volume types I/O performance scales with size (here, for every terabyte of requested storage server would request 100m CPU and 250m memory)
The statefulset for the provisioner, instead of maintaining a mapping from PVs to subpaths, would maintain a mapping from PVs to servers.
Summary of advantages over current approach:
I'm relatively new to the CSI ecosystem but I'm wondering if something like this is feasible/has been thought about.
Is there the possibility to add Network Policies to avoid other pods in a multi-tenant cluster from accessing Ganesha server without impacting volume mounts ?
Hi there,
apologies if I am missing out something obvious.
I am using minikube to test out the NFS server and following the instructions and using the deployment:
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-provisioner
---
kind: Service
apiVersion: v1
metadata:
name: nfs-provisioner
labels:
app: nfs-provisioner
spec:
ports:
- name: nfs
port: 2049
- name: nfs-udp
port: 2049
protocol: UDP
- name: nlockmgr
port: 32803
- name: nlockmgr-udp
port: 32803
protocol: UDP
- name: mountd
port: 20048
- name: mountd-udp
port: 20048
protocol: UDP
- name: rquotad
port: 875
- name: rquotad-udp
port: 875
protocol: UDP
- name: rpcbind
port: 111
- name: rpcbind-udp
port: 111
protocol: UDP
- name: statd
port: 662
- name: statd-udp
port: 662
protocol: UDP
selector:
app: nfs-provisioner
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: nfs-provisioner
spec:
selector:
matchLabels:
app: nfs-provisioner
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-provisioner
spec:
serviceAccountName: nfs-provisioner
containers:
- name: nfs-provisioner
image: quay.io/kubernetes_incubator/nfs-provisioner:latest
ports:
- name: nfs
containerPort: 2049
- name: nfs-udp
containerPort: 2049
protocol: UDP
- name: nlockmgr
containerPort: 32803
- name: nlockmgr-udp
containerPort: 32803
protocol: UDP
- name: mountd
containerPort: 20048
- name: mountd-udp
containerPort: 20048
protocol: UDP
- name: rquotad
containerPort: 875
- name: rquotad-udp
containerPort: 875
protocol: UDP
- name: rpcbind
containerPort: 111
- name: rpcbind-udp
containerPort: 111
protocol: UDP
- name: statd
containerPort: 662
- name: statd-udp
containerPort: 662
protocol: UDP
securityContext:
capabilities:
add:
- DAC_READ_SEARCH
- SYS_RESOURCE
privileged: true
args:
- "-provisioner=example.com/nfs"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: SERVICE_NAME
value: nfs-provisioner
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
imagePullPolicy: "Always"
volumeMounts:
- name: export-volume
mountPath: /export
volumes:
- name: export-volume
hostPath:
path: /tmp/export
and rbac:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services", "endpoints"]
verbs: ["get"]
- apiGroups: ["extensions"]
resources: ["podsecuritypolicies"]
resourceNames: ["nfs-provisioner"]
verbs: ["use"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-provisioner
subjects:
- kind: ServiceAccount
name: nfs-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: ClusterRole
name: nfs-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-provisioner
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-provisioner
subjects:
- kind: ServiceAccount
name: nfs-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: Role
name: leader-locking-nfs-provisioner
apiGroup: rbac.authorization.k8s.io
In ganesha.log I am getting:
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] main :MAIN :EVENT :nfs-ganesha Starting: Ganesha Version 2.8.2
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] claim_posix_filesystems :FSAL :CRIT :Could not stat directory for path /nonexistent
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] vfs_create_export :FSAL :CRIT :resolve_posix_filesystem(/nonexistent) returned No such file or directory (2)
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL VFS
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/nonexistent) to (/nonexistent)
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] export_commit_common :CONFIG :CRIT :Export id 0 can only export "/" not (/nonexistent)
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!!
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): 1 validation errors in block FSAL
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): Errors processing block (FSAL)
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): 1 validation errors in block EXPORT
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): Errors processing block (EXPORT)
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/usr/local/var/run/ganesha) already exists
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:22)
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
04/12/2020 17:48:18 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[reaper] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE
Would appreciate any pointers
In #56 @kvaps suggests to let two dockerfiles be the same Dockerfile. Excellent!
I'm not sure about the status of the arm Dockerfile, but trialing to build the x86_64 arm dockerfile on my arm64 based Raspberry PI 4, it seem to work fine as well.
Perhaps there should simply be one single Dockerfile instead of three or two?
Add option for additional annotations at the nfs StorageClass object.
For example to be used when configuring Kasten (K10) backup/restore solution.
Issue was reported by @albanm on the older repo at: kubernetes-retired/external-storage#1184
A proposed solution by @linki for this is available at: kubernetes-retired/external-storage#1352
Copy past of the issue description.
Describe the bug
The container restarts frequently when the api server is not responding but it should not have to do it.
What happened:
I have a cluster with latency and sometimes failure issues of the api server. I have submitted a bug to my cloud provider that manages the control plane.
My issue with nfs-server-provisioner is that during periods of failures of the api server I seel logs like this and the container restarts :
I0619 07:05:11.965197 1 leaderelection.go:231] failed to renew lease nfs/cluster.local-nfs-data-nfs-server-provisioner: failed to tryAcquireOrRenew context deadline exceeded
F0619 07:05:11.966172 1 controller.go:646] leaderelection lost
I have lots of problems with the PVCs based on this provisionner: binding timeouts, lost file writes, etc.
I am not certain but I think it is related to these frequent restarts of the nfs server.
What you expected to happen:
The nfs server should tolerate api server failures.
How to reproduce it (as minimally and precisely as possible):
I do not manage the control plane of my cluster. But I suppose that manually stopping the api server should recreate the issue.
Anything else we need to know:
I understand restarting the container in this case is probably the expected behavior, but I think it is wrong.
replication is not relevant in the case of nfs-server-provisioner and I don't see the point of a leader election
stability issues of the api server should not propagate to a storage provisioner and then to a bunch of volumes used by a bunch of services
I have deployed the NFS service and it works when volumes are automatically created with the volumeClaimTemplate. An NFS share is generated and all is fine. However, when I create a PVC and PV, it does not seem to generate the relative NFS share.
Storage class
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: mynfs-nfs
mountOptions:
- vers=4.1
provisioner: mynfs/nfs
reclaimPolicy: Delete
volumeBindingMode: Immediate
PV (create with and without provisioner_id, no change)
apiVersion: v1
kind: PersistentVolume
metadata:
name: storage
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 30Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: pvc-storage
namespace: shares
nfs:
path: /export/storage
server: 10.158.1.1
persistentVolumeReclaimPolicy: Retain
storageClassName: mynfs-nfs
volumeMode: Filesystem
PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-storage
namespace: shares
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi
storageClassName: mynfs-nfs
volumeMode: Filesystem
volumeName: storage
I did notice that the EXPORT section is not present when getting the yaml details of the volume, and also vfs.conf is not updated.
*** EDIT ***
The path mentioned above, /export/storage is also not created. It seems the whole folder, and export initialization is skipped.
Hey, would love to see support for the path config to be able to be set for consistent volume naming/grouping on the filesystem as with the nfs-subdir provisioner.
pathPattern: "$${.PVC.namespace}/$${.PVC.name}"
Hello
The version of NFS - Ganesha used in this provisionner is the version 2.8.2 which is old (Jully 2019 according to its repository.
Could you please upgrade the version to 3.5, latest GA available ?
Best regards
Failed to pull image "k8s.gcr.io/sig-storage/nfs-provisioner:v3.0.0": rpc error: code = Unknown desc = Error choosing image instance: no image found in manifest list for architecture arm64, variant "v8", OS linux
Problem:
I am trying to get nfs-server-provisioner
to work in K3D. PVCs and PVs are handled correctly, but Pods cant mount the volume referenced by a PVC. I created an example to make it easy to reproduce the issue in a GitRepo: https://github.com/tricky42/k3d-nfs with a more detailed description.
In this setup the local-path
provider from K3D is used to provide the PV for the NFS server automatically through the PVC issued by the nfs-server-provisioner
. PVCs of storage class nfs
can be provisioned (status: bound), but still pods trying to mount volumes which reference a PVC of storageClass nfs
(in this example the PVC nfs-for-pods
) are not starting up due to:
MountVolume.SetUp failed for volume "pvc-52c0d57a-b285-45b6-b04c-41175604b0d0" : mount failed: exit status 255
Mounting command: mount Mounting arguments: -t nfs -o vers=3 10.43.192.199:/export/pvc-52c0d57a-b285-45b6-b04c-
41175604b0d0 /var/lib/kubelet/pods/c9e21798-76ad-404d-b089-c9fff88e0a86/volumes/kubernetes.io~nfs/pvc-
52c0d57a-b285-45b6-b04c-41175604b0d0 Output: mount: mounting 10.43.192.199:/export/pvc-52c0d57a-b285-45b6-
b04c-41175604b0d0 on /var/lib/kubelet/pods/c9e21798-76ad-404d-b089-c9fff88e0a86/volumes/kubernetes.io~nfs/pvc-
52c0d57a-b285-45b6-b04c-41175604b0d0 failed: Connection refused
Unable to attach or mount volumes: unmounted volumes=[nfs-mount], unattached volumes=[nfs-mount default-token
-t7qxd]: timed out waiting for the condition
To quickly reproduce this, just enter:
$ make all
So far I couldnt find out, why the Pod cant mount the volume managed by the nfs server :(
Hi! I tried to use nfs-provisioner with openshift.
But pods are failing when are allocated in a different node where the nfs-provisioner pod is running. I suppose that it is because CLUSTERIP service is not accessible from pods in different nodes.
Is there another way to expose the provisioner rather than using clusterIP?
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
item-onboard-test 0/1 ContainerCreating 0 2m18s <none> jorge-thks6-worker-eastus1-pc27r <none> <none>
item-onboard-test-85c9549869-khfnn 1/1 Running 0 106s 10.129.2.17 jorge-thks6-worker-eastus2-kgcf8 <none> <none>
item-onboard-test-x7gz2 0/1 ContainerCreating 0 100s <none> jorge-thks6-worker-eastus3-xjn78 <none> <none>
nfs-provisioner-7d898f4fdd-tn6ws 1/1 Running 0 8m37s 10.129.2.14 jorge-thks6-worker-eastus2-kgcf8 <none> <none>
test 1/1 Running 0 2m45s 10.129.2.16 jorge-thks6-worker-eastus2-kgcf8 <none> <none>
test-no-pvc 1/1 Running 0 8m36s 10.129.2.13 jorge-thks6-worker-eastus2-kgcf8 <none> <none>
Many thanks,
Jorge
The yaml doesn't match the text.
https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner#quickstart
$ docker pull quay.io/external_storage/nfs-ganesha-server-and-provisioner:latest
Error response from daemon: unauthorized: access to the requested resource is not authorized
kubectl get deploy nfs-provisioner -o yaml | sed "s|quay.io/external_storage/nfs-ganesha-server-and-provisioner:latest|k8s.gcr.io/sig-storage/nfs-provisioner:v3.0.0|" | kubectl replace -f -
$ k version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:58:47Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:52:18Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}
$ k applay -f deploy/kubernetes/write-pod.yaml
3m15s Normal Scheduled pod/write-pod Successfully assigned default/write-pod to k8s-worker04.kubernetes.lan
3m14s Normal Pulled pod/write-pod Container image "gcr.io/google_containers/busybox:1.24" already present on machine
3m14s Normal Created pod/write-pod Created container write-pod
3m14s Normal Started pod/write-pod Started container write-pod
3m12s Warning FailedMount pod/write-pod MountVolume.SetUp failed for volume "kube-api-access-b9tt8" : object "default"/"kube-root-ca.crt" not registered
$ k get pod
NAME READY STATUS RESTARTS AGE
nfs-provisioner-5b485c6b65-96t5r 1/1 Running 0 14m
read-pod 0/1 Completed 0 8m41s
write-pod 0/1 Completed 0 8m48s
As Helm has developed over time, various best practices has been updated. I'd be happy to contribute PRs that would make this Helm chart up to date with such practices. But, I'd like some input with regards to if that would be wanted, and if it could be okay to make some breaking changes at this time.
I figure since this Helm chart isn't being published right now, it would be suitable to do breaking changes now before it becomes published again, for example at https://artifacthub.io/ which is the new Helm Hub location.
app
it would be app.kubernetes.io/name
(like discussed for this Helm chart: jupyterhub/zero-to-jupyterhub-k8s#1867).helm3 upgrade
even though it was quite easy with helm2 upgrade --force
, so if it is ever to be done, it's good to do sooner rather than later in my mind.apiVersion: 2
.{{- with ... }}
instead of {{- if }}
when it makes senseinclude
instead of template
.nindent
with whitespace chomping left instead of indent
when using toYaml
Would you be open to such breaking changes and/or such non breaking changes to the Helm chart?
I'm maintaining the JupyterHub Helm chart and have spent a lot of time working with k8s and Helm chart the last ~3 years. I'm motivated to contribute to this project as I think it serves a very important piece in the infrastructure I want to help many others make use of, for example universities that want to provide a compute platform for their researchers and students with personal home directories.
Raising this tracker issue to pull in the changes from kubernetes-retired/external-storage#1320 (cc: @sopak )
It seems that the nfs server is not automatically reloaded when the mountoptions are changed.. Wouldn't it be good to f.e. set an annotation with a sha256 hash of the mount options?
I have been testing the helm chart in order to deploy the nfs provisioner because we would like to use it for production for many dynamic temporary PVs.
All the default values works fine but it creates only one replica of the provisionner (and the server).
For my use case I need to have more relyability and may be more throughput and was hoping to be able to increase the number of replicas.
1/ So I tried to set the replicaCount=2
. Both replicas started fine. But then we I try to claim a persistent volume it failed.
This is the error from the replica-0
controller.go:908] error syncing claim "36caa83a-31f9-4875-bc1d-b4656e06895e": failed to provision volume with StorageClass "nfs":
here is the error in the of the pvc
Warning ProvisioningFailed 38s (x4 over 2m23s) cluster.local/nfs-nfs-server-provisioner_nfs-nfs-server-provisioner-0_09128ba9-3d58-46eb-bcc2-6318389bc26e failed to provision volume with StorageClass "nfs": error getting NFS server IP for volume: service SERVICE_NAME=nfs-nfs-server-provisioner is not valid; check that it has for ports map[{111 TCP}:true {111 UDP}:true {662 TCP}:true {662 UDP}:true {875 TCP}:true {875 UDP}:true {2049 TCP}:true {2049 UDP}:true {20048 TCP}:true {20048 UDP}:true {32803 TCP}:true {32803 UDP}:true] exactly one endpoint, this pod's IP POD_IP=10.223.67.159
I figured that because it us using a single service in front of a 2 pods then the loadbalancing cannot make this setup work. ( maybe headless service would be more appropriate).
2/ So I have decided to alter the helm chart to disable the SERVICE_NAME
environment variable by commenting this. I have seen in the code that the provisioner would use the pod IP so why not give it a try.
So this works better as the PV managed to be created. but...
But it seems that only the first replica is managing the PV and the second replica is simply ignored.
Is there a way to make this work ? I am not familiar with the provider mechanism but is it possible that multiple instances may manage different PVs ?
Sorry for this long post, I hope that my questions are clean enough.
Hey,
I have installed the provider through helm:
helm install data-nfs-server-provisioner stable/nfs-server-provisioner -f nvalues.yaml --namespace clearml
Then I have added a Persistent Volume:
apiVersion: v1
kind: PersistentVolume
metadata:
name: data-nfs-server-provisioner-0
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteOnce
gcePersistentDisk:
fsType: "ext4"
pdName: "clearml-cache"
claimRef:
namespace: clearml
name: data-data-nfs-server-provisioner-0
While clearml-cache
was created with:
$ gcloud compute disks create --size=100GB --zone=$ZONE --project "$PROJECT_ID" clearml-cache
Then I have added a PV and PVC:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: clearml-agent-cachingt-claim
namespace: clearml
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 100G
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: clearml-agent-cachingt
namespace: clearml
spec:
capacity:
storage: 100G
storageClassName: "nfs"
accessModes:
- ReadWriteMany
nfs:
path: /
server: data-nfs-server-provisioner.clearml.svc.cluster.local
claimRef:
namespace: clearml
name: clearml-agent-cachingt-claim
EOF
and a pod:
apiVersion: v1
kind: Pod
metadata:
name: nfs-shaked-test
spec:
securityContext:
runAsUser: 0
runAsGroup: 0
fsGroup: 0
containers:
- name: nfs-shaked-test
image: d3fk/nfs-client
command: [ "sleep" ]
args: [ "infinity" ]
securityContext:
privileged: true
resources: {}
env:
- name: SHARE
value: "/shakedsnfs"
- name: SERVER
value: "data-nfs-server-provisioner.clearml.svc.cluster.local"
volumeMounts:
- mountPath: /shakedsnfs
name: caching-nfs
readOnly: false
volumes:
- name: caching-nfs
persistentVolumeClaim:
claimName: clearml-agent-cachingt-claim
I tried to set this pod with and without some of the above configuration, for example with and without privileged
, runAs
, fsGroup
and readOnly: false
. I have also tried to use the write-pod example but I keep seeing the following error:
/ # touch shakedsnfs/1
touch: shakedsnfs/1: Read-only file system
I have found nothing in the logs:
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] main :MAIN :EVENT :nfs-ganesha Starting: Ganesha Version 2.8.2
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] claim_posix_filesystems :FSAL :CRIT :Could not stat directory for path /nonexistent
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] vfs_create_export :FSAL :CRIT :resolve_posix_filesystem(/nonexistent) returned No such file or directory (2)
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL VFS
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/nonexistent) to (/nonexistent)
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] export_commit_common :CONFIG :CRIT :Export id 0 can only export "/" not (/nonexistent)
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!!
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): 1 validation errors in block FSAL
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): Errors processing block (FSAL)
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): 1 validation errors in block EXPORT
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): Errors processing block (EXPORT)
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/usr/local/var/run/ganesha) already exists
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:22)
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
14/05/2021 19:10:38 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[reaper] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE
16/05/2021 17:20:18 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 5962, old: 5961; deq new: 5961, old: 5961
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] main :MAIN :EVENT :nfs-ganesha Starting: Ganesha Version 2.8.2
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] claim_posix_filesystems :FSAL :CRIT :Could not stat directory for path /nonexistent
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] vfs_create_export :FSAL :CRIT :resolve_posix_filesystem(/nonexistent) returned No such file or directory (2)
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL VFS
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/nonexistent) to (/nonexistent)
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] export_commit_common :CONFIG :CRIT :Export id 0 can only export "/" not (/nonexistent)
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!!
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): 1 validation errors in block FSAL
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): Errors processing block (FSAL)
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): 1 validation errors in block EXPORT
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): Errors processing block (EXPORT)
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/usr/local/var/run/ganesha) already exists
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:22)
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
16/05/2021 21:02:26 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[reaper] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] main :MAIN :EVENT :nfs-ganesha Starting: Ganesha Version 2.8.2
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] claim_posix_filesystems :FSAL :CRIT :Could not stat directory for path /nonexistent
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] vfs_create_export :FSAL :CRIT :resolve_posix_filesystem(/nonexistent) returned No such file or directory (2)
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL VFS
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/nonexistent) to (/nonexistent)
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] export_commit_common :CONFIG :CRIT :Export id 0 can only export "/" not (/nonexistent)
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!!
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): 1 validation errors in block FSAL
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): Errors processing block (FSAL)
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): 1 validation errors in block EXPORT
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): Errors processing block (EXPORT)
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/usr/local/var/run/ganesha) already exists
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:22)
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
16/05/2021 22:24:13 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[reaper] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE
kubectl logs -f data-nfs-server-provisioner-0
:
I0516 22:22:43.183805 1 main.go:64] Provisioner cluster.local/data-nfs-server-provisioner specified
I0516 22:22:43.183891 1 main.go:88] Setting up NFS server!
I0516 22:22:43.401571 1 server.go:149] starting RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0516 22:22:43.401613 1 server.go:160] ending RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0516 22:22:43.414202 1 server.go:134] Running NFS server!
I assume I am missing something, but honestly I can't seem to find it.
Any chance you could help?
Thank you
Shaked
Hello,
I believe there is something probably off with the latest nfs server provisioner chart.
My kubernetes version:
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-21T23:01:33Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
Steps I did:
$ helm repo add nfs-ganesha-server-and-external-provisioner https://kubernetes-sigs.github.io/nfs-ganesha-server-and-external-provisioner/
$ helm install my-release nfs-ganesha-server-and-external-provisioner/nfs-server-provisioner
Why the pod keeps restarting? Looking at the logs, it exits with return code 137
kl my-release-nfs-server-provisioner-0
I1022 01:44:14.624283 1 main.go:65] Provisioner cluster.local/my-release-nfs-server-provisioner specified
I1022 01:44:14.631565 1 main.go:89] Setting up NFS server!
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Fri, 22 Oct 2021 12:41:43 +1100
Finished: Fri, 22 Oct 2021 12:42:48 +1100
I was digging into the github action that tests the chart, I am wondering if the workflow did what it was supposed to do:
Some steps takes 0s ? Is that expected ?
Thank you.
Cheers.
I note that there is a set of yaml files that i assume to mirror the Helm chart, but they are not updated as work is done on the Helm chart.
Should they be maintained at all, and if so, how should they be maintained? By using helm template
to generate a rendered set of resources from the Helm charts default values perhaps?
It's my understanding that the project currently doesn't support creating a highly-available NFS server cluster given the lack of examples on the subject. However, I do also see the leader-elect
flag on the binary with the description Start a leader election client and gain leadership before executing the main loop. Enable this when running replicated components for high availability. Default false.
.
I'm wondering if anyone has used the leader-elect
in a highly available NFS cluster configuration and what the setup for that looked like.
From an old presentation on NFS-Ganesha here, it seems that NFS-Ganesha requires some coordination to handle failover well. I suspect this would be set up manually by any user wanting to use a highly available NFS cluster setup with this provisioner, but I'm not certain.
Going off of the assumption that this project currently can't set up a multiple-NFS-Server cluster that is highly available today, is it within the purview of this project to support such a thing and accept contributions to design and implement the ability to set up a highly-available NFS cluster inside Kubernetes?
Hello!
We're playing around with nfs-ganesha-server-and-external-provisioner
, but our server/provisioner pod is going into ImagePullBackOff:
Warning Failed 9m48s (x4 over 11m) kubelet, ip-10-100-89-6.us-west-2.compute.internal Failed to pull image "quay.io/external_storage/nfs-ganesha-server-and-provisioner:latest": rpc error: code = Unknown desc = Error response from daemon: unauthorized: access to the requested resource is not authorized
Warning Failed 9m48s (x4 over 11m) kubelet, ip-10-100-89-6.us-west-2.compute.internal Error: ErrImagePull
Warning Failed 9m36s (x6 over 11m) kubelet, ip-10-100-89-6.us-west-2.compute.internal Error: ImagePullBackOff
Normal BackOff 76s (x41 over 11m) kubelet, ip-10-100-89-6.us-west-2.compute.internal Back-off pulling image "quay.io/external_storage/nfs-ganesha-server-and-provisioner:latest"
pulling the image from the command line doesn't seem to work either, it looks like the image just isn't there on quay.io at all:
$ docker pull quay.io/external_storage/nfs-ganesha-server-and-external-provisioner:latest
Error response from daemon: unauthorized: access to the requested resource is not authorized
I don't mind building the image ourselves, but the docs should probably be updated to state that it's not available on quay just yet :)
Cheers!
Hi,
In my point.
I have installed nfs-ganesha-server-and-external-provisioner by helm.
After deploy app and create PV we have working app and etc.
But when autoscaling or updating node nfs-server recreate and we have empty folder export
How i can save data after reinstall nfc server/
Thx
There are no releases for this project. It looks some pull requests in the prior read-only project were not pulled across into this version. It is hard to tell what version this corresponds to as there are no tags or releases. The helm chart included in this repo is not versioned either (typically there's a branch that communicates the releases to third party helm chart hubs).
The Quay repository referenced in the helm chart shows version 2.3.0, but then nowhere is a tag or release that corresponds to it.
I asked @yuvipanda whom I learn a lot of NFS matters from about this:
nfs-ganesha-server-and-external-provisioner/deploy/helm/values.yaml
Lines 74 to 75 in 027c2eb
Yeah, I think the default should be 4.2 since nfs-ganesha supports it, unless they have a strong reason to not do that.
Should we transition to suggest 4.2 by default instead?
I'm not sure what this file does, but I see no mention of it in docs and it lacks comments etc. I believe it is a Kubernetes config on how to access a k8s cluster, and if it is that, it wouldn't make sense to locate in the deploy/docker folder.
nfs-ganesha-server-and-external-provisioner/deploy/docker/config
Lines 1 to 15 in 027c2eb
We are using the above code and running it with the default setup. We are using an Azure Premium disk as the backing disk. When we write a lot of IO to the volume from the client pod, we see the memory on the provisioner pod growing.
has anyone seen this before?
The documented helm install command:
helm install stable/nfs-server-provisioner
gives a deprecated warning as the helm stable repo has been deprecated. Is there a plan to move this to a different helm repo?
Please add a field to allow us to specify pod annotations.
Example use case: disallowing pod eviction by Cluster Autoscaler:
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
reproduce:
docker stop nfs-ganesha
(or restart host os)
docker start nfs-ganesha
Container goes into infinite restart loop with error message:
Error setting up NFS server: dbus-daemon failed with error: exit status 1, output: dbus-daemon[18]: Failed to start message bus: The pid
file "/var/run/dbus/pid" exists, if the message bus is not running, remove this file
Hello,
We've had a disaster recovery on our cluster and we had to re-provision the nfs server provisioner.
When we reprovisioned the server, it created a new service which got a new IP address, which made all the PVs invalid.
We restored the functionality by adding a new ad-hoc service in which we specified the service IP to correspond to the PVs ones. We now need this wart in the cluster.
I was wondering if we could use the nfs service name instead of the IP when provisioning the PVs ?
Right now, when the PV is created, the IP is placed:
Source:
Type: NFS (an NFS mount that lasts the lifetime of a pod)
Server: 10.222.45.230
Path: /export/pvc-fd579ce4-6203-4eb8-a5eb-30436d8cf52f
ReadOnly: false
See here:
where we return the service IP instead of the name.I'm quite sure there's a good reason for doing it via IP, but this is limiting when there's a disaster recoveery. Any idea/comment ?
Running ganesha in k8s.
Getting the following error when trying to create PV:
I0406 00:26:56.616749 1 main.go:65] Provisioner example.com/nfs specified
I0406 00:26:56.617163 1 main.go:89] Setting up NFS server!
I0406 00:26:57.573661 1 server.go:149] starting RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0406 00:26:57.573762 1 server.go:160] ending RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0406 00:26:57.575869 1 server.go:134] Running NFS server!
E0406 00:27:02.889823 1 controller.go:908] error syncing claim "2937d4b7-2215-4d84-be2d-5552790e7cd4": failed to provision volume with StorageClass "example-nfs": error getting NFS server IP for volume: service SERVICE_NAME=nfs-provisioner is not valid; check that it has for ports map[{111 TCP}:true {111 UDP}:true {662 TCP}:true {662 UDP}:true {875 TCP}:true {875 UDP}:true {2049 TCP}:true {2049 UDP}:true {20048 TCP}:true {20048 UDP}:true {32803 TCP}:true {32803 UDP}:true] exactly one endpoint, this pod's IP POD_IP=10.42.2.26
Here is the sum of my config:
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: ganesha
name: nfs-provisioner
spec:
selector:
matchLabels:
app: nfs-provisioner
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-provisioner
spec:
nodeSelector:
purpose: nas
serviceAccount: nfs-provisioner
containers:
- name: nfs-provisioner
image: k8s.gcr.io/sig-storage/nfs-provisioner:v3.0.0
ports:
- name: nfs
containerPort: 2049
- name: nfs-udp
containerPort: 2049
protocol: UDP
- name: nlockmgr
containerPort: 32803
- name: nlockmgr-udp
containerPort: 32803
protocol: UDP
- name: mountd
containerPort: 20048
- name: mountd-udp
containerPort: 20048
protocol: UDP
- name: rquotad
containerPort: 875
- name: rquotad-udp
containerPort: 875
protocol: UDP
- name: rpcbind
containerPort: 111
- name: rpcbind-udp
containerPort: 111
protocol: UDP
- name: statd
containerPort: 662
- name: statd-udp
containerPort: 662
protocol: UDP
securityContext:
capabilities:
add:
- DAC_READ_SEARCH
- SYS_RESOURCE
args:
- "-provisioner=example.com/nfs"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: SERVICE_NAME
value: nfs-provisioner
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: export-volume
mountPath: /export
volumes:
- name: export-volume
hostPath:
path: /mnt/data
---
apiVersion: v1
kind: Namespace
metadata:
name: ganesha
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
namespace: ganesha
name: nfs
spec:
storageClassName: example-nfs
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Mi
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: ganesha
name: nfs-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services", "endpoints"]
verbs: ["get"]
- apiGroups: ["extensions"]
resources: ["podsecuritypolicies"]
resourceNames: ["nfs-provisioner"]
verbs: ["use"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: ganesha
name: run-nfs-provisioner
subjects:
- kind: ServiceAccount
name: nfs-provisioner
# replace with namespace where provisioner is deployed
namespace: ganesha
roleRef:
kind: ClusterRole
name: nfs-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-provisioner
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-provisioner
subjects:
- kind: ServiceAccount
name: nfs-provisioner
namespace: ganesha
roleRef:
kind: Role
name: leader-locking-nfs-provisioner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: ganesha
name: nfs-provisioner
kind: Service
apiVersion: v1
metadata:
namespace: ganesha
name: nfs-provisioner
labels:
app: nfs-provisioner
spec:
ports:
- name: nfs
port: 2049
targetPort: 2049
protocol: TCP
- name: nfs-udp
port: 2049
targetPort: 2049
protocol: UDP
- name: nlockmgr
port: 32803
targetPort: 32803
protocol: TCP
- name: nlockmgr-udp
port: 32803
targetPort: 32803
protocol: UDP
- name: mountd
port: 20048
targetPort: 20048
protocol: TCP
- name: mountd-udp
port: 20048
targetPort: 20048
protocol: UDP
- name: rquotad
port: 875
targetPort: 875
protocol: TCP
- name: rquotad-udp
port: 875
targetPort: 875
protocol: UDP
- name: rpcbind
port: 111
targetPort: 111
protocol: TCP
- name: rpcbind-udp
port: 111
targetPort: 111
protocol: UDP
- name: statd
port: 662
targetPort: 662
protocol: TCP
- name: statd-udp
port: 663
targetPort: 663
protocol: UDP
selector:
app: nfs-provisioner
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
namespace: ganesha
name: example-nfs
provisioner: example.com/nfs
mountOptions:
- vers=4.1
---
apiVersion: v1
kind: Pod
metadata:
namespace: ganesha
name: write-pod
spec:
containers:
- name: write-pod
image: gcr.io/google_containers/busybox:1.24
command:
- "/bin/sh"
args:
- "-c"
- "touch /mnt/SUCCESS && exit 0 || exit 1"
volumeMounts:
- name: nfs-pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: nfs-pvc
persistentVolumeClaim:
claimName: nfs
Hi, Is it possible to use this to mount the pvc as a specific user? I see we can mount with a certain gid using that parameter, but then the uid is some random number. There is no parameter option for uid as I've tried, so is there some other way to mount an nfs pvc with a specific uid?
The Priority Class section is causing issues in version 1.4.0.
If priorityClass.create is false and priorityClass.name is "", priorityClassName is set to nil (see below)
Error: template: nfs-server-provisioner/templates/statefulset.yaml:120:24: executing "nfs-server-provisioner/templates/statefulset.yaml" at <.Values.priorityClass.name>: nil pointer evaluating interface {}.name
If priorityClass.create is set to true and priorityclass.name is set to "default", the error is as shown below.
Error: error validating "": error validating data: ValidationError(PriorityClass): missing required field "value" in io.k8s.api.scheduling.v1.PriorityClass
Consider regenerating ganesha exports config file /export/vfs.conf
every time at startup.
the export block already can be found in allocated PVs:
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
EXPORT_block: "\nEXPORT\n{\n\tExport_Id = 1;\n\tPath = /export/pvc-40a3866c-83d6-47e9-ac9f-6d0462a4ab1d;\n\tPseudo
= /export/pvc-40a3866c-83d6-47e9-ac9f-6d0462a4ab1d;\n\tAccess_Type = RW;\n\tSquash
= no_root_squash;\n\tSecType = sys;\n\tFilesystem_id = 1.1;\n\tFSAL {\n\t\tName
= VFS;\n\t}\n}\n"
Export_Id: "1"
Project_Id: "0"
Provisioner_Id: 82f81614-14e4-474a-b852-facc2906d3ab
kubernetes.io/createdby: nfs-dynamic-provisioner
pv.kubernetes.io/provisioned-by: cluster.local/nextcloud-nfs-server-provisioner
name: pvc-40a3866c-83d6-47e9-ac9f-6d0462a4ab1d
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: nextcloud-nfs
namespace: nextcloud
resourceVersion: "331724805"
uid: 40a3866c-83d6-47e9-ac9f-6d0462a4ab1d
mountOptions:
- vers=3
nfs:
path: /export/pvc-40a3866c-83d6-47e9-ac9f-6d0462a4ab1d
server: 10.96.15.81
persistentVolumeReclaimPolicy: Delete
storageClassName: nextcloud-nfs
When deploying using
git clone https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner
/tmp/nfs-ganesha-server-and-external-provisioner/deploy/helm
git checkout V3.0.0
helm install nfs-server-provisioner . \
--namespace nfs-server-provisioner --create-namespace \
--set persistence.storageClass="openebs-hostpath" \
--set persistence.size="10Gi" \
--set persistence.enabled=true
The k8s instance become unresponsible with a high cpu load.
Not even sure where to start diagnosing this issue.
Any ideas?
top - 07:15:51 up 25 min, 3 users, load average: 28.98, 45.59, 28.49
Tasks: 247 total, 2 running, 245 sleeping, 0 stopped, 0 zombie
%Cpu(s): 38.0 us, 15.5 sy, 0.0 ni, 28.7 id, 17.3 wa, 0.3 hi, 0.2 si, 0.0 st
MiB Mem : 31980.4 total, 2275.2 free, 28775.4 used, 929.8 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 2822.9 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
33770 root 20 0 1101992 376880 75920 S 105.6 1.2 0:04.78 kube-apiserver
32624 root 20 0 52.0g 27.0g 3392 R 99.3 86.6 0:09.76 rpcinfo
k8s version: 1.21
os: fedora coreos
Hi,
Using this provisioner works fine when pods share files from storage obtained using the same namespace/PVC as the shared folder name is dynamically created based on namespace/PVC/PV.
How can I manage to share files between two namespaces ?
Regards
Mic
Hi, are one of these listed helm charts from this repo?
https://artifacthub.io/packages/search?page=1&ts_query_web=nfs-server-provisioner
If not, this would be good to add.
We have a kubernetes-sigs org there already - let me know if you'd like me to add you.
Current image doesn't run on newer Macbooks:
Failed to pull image "k8s.gcr.io/sig-storage/nfs-provisioner:v3.0.0": rpc error: code = N │
│ otFound desc = failed to pull and unpack image "k8s.gcr.io/sig-storage/nfs-provisioner:v3.0.0": no match for platform in manifest: not found
Any chance ARM variant of this image can be added to the build?
Thanks!
I'm currently using a handmade NFS volume solution and would like to move to the official NFS image.
Is it possible to take the existing PV with me to the new provisionier solution? Or is there a way to copy the files from the old disk to the new one?
In the nfs-ganesha project, we have this issue raised when Export_Id 152 is provisioned:
I am wondering why the provisioning is specifying Filesystem_Id? It is not recommended to use this export configuration option for nfs-ganesha and instead allow the underlying filesystem to provide the filesystem ID via a UUID or by the device ID the filesystem is mounted on. There are situations where a stable Filesystem_Id is not available and the option must be specified. One such place is FSAL_PSEUDO exports, in particular the automatically configured FSAL_PSEUDO export for the NFSv4 PseudoFS root. This auto-configured export (Export_Id 0) uses Filesystem_Id 152.152. This is a long standing configuration predating the ability to use UUID or Device ID from the underlying filesystem. The use of Export_Id.Export_Id for the provisioned exports Filesystem_Id is a problem if Export_Id 152 is used.
If you could either change the Filesystem_Id or eliminate the option, that would be ideal. If not, the best solution would then be to consider Export_Id 152 to also be reserved.
Another option would be for the provisioner to explicitly configure the PseduoFS root and specify a Filesystem_Id that doesn't conflict with the Export_Id.Export_Id ideal. I'm not sure if 0.0 is ok to use, but any X.Y where X and Y are not the same would work.
It might still relate to kubernetes-retired/external-storage#1142.
and when I tried to resize the volume it shows this message on the describe
command:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ExternalExpanding 42s (x3 over 12h) volume_expand Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
Is there any plan to support this feature?
NetworkPolicy resources can help pods avoid doing things they shouldn't do by just allowing what needs to be allowed. By describing explicit needs of network communication within the resources of this Helm chart, all other network communication can be rejected as a level of protection.
With addition of non-amd64 platforms, now prow job is taking more time to finish building the container images because of the cross-build(qemu simulation mode) and times out for default timeout(2hrs)
Sample job run:
https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-nfs-ganesha-server-and-external-provisioner-push-images/1459900631392194560
something like this is needed to fix the timeout issue:
https://github.com/kubernetes/test-infra/blob/9299b9708f6c5c15e8a41c2f2b989033f6d239b0/config/jobs/kubernetes/sig-network/sig-network-misc.yaml#L13:L15,
I've been doing some testing and with NFSv4.2 pvs provisioned do not support xattr.
But they should by my understanding.
lgetxattr("/data/upper/.", "system.posix_acl_access", 0x196d920, 4096) = -1 EOPNOTSUPP (Operation not supported)
writev(6, [{iov_base="\20\0\0\0\241\377\377\377\22\0\0\0\0\0\0\0", iov_len=16}], 1) = 16
splice(6, NULL, 8, NULL, 135168, 0) = 72
Any ideas?
Right now we are trying to figure out how to install this chart in a dedicated namespace?
But it looks like it's not possible when I look at the chart. Is there any reason for this, so that one can specify one?
Our helmfile looks like following:
environments:
production:
staging:
repositories:
- name: kvaps
url: https://kvaps.github.io/charts
releases:
- name: {{ .Environment.Name }}
chart: charts/company
values:
- charts/company/configs/{{ .Environment.Name }}.yaml
set:
- name: railsmasterkey
file: config/master.key
- name: nfs-server-provisioner
version: 1.2.0
chart: kvaps/nfs-server-provisioner
namespace: {{ .Environment.Name }} <--- this is not working?
PVC is not getting provisioned on some systems. It works on some and not on others. This is the error in the PVC
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 6m4s mediation.apps.nfs.mycompany.nfs.provisioner.io_mediation-apps-nfs-mycompany-nfs-provisioner-66fd6b784-lp6dq_d33c2841-0caf-4b29-b087-61e3fbe9a24e failed to provision volume with StorageClass "mediation-apps-nfs-mycompany-nfs-provisioner": error getting NFS server IP for volume: service SERVICE_NAME=mediation-apps-nfs-mycompany-nfs-provisioner is not valid; check that it has for ports map[{111 TCP}:true {111 UDP}:true {662 TCP}:true {662 UDP}:true {875 TCP}:true {875 UDP}:true {2049 TCP}:true {2049 UDP}:true {20048 TCP}:true {20048 UDP}:true {32803 TCP}:true {32803 UDP}:true] exactly one endpoint, this pod's IP POD_IP=10.42.0.151
Normal Provisioning 2m19s (x5 over 6m4s) mediation.apps.nfs.mycompany.nfs.provisioner.io_mediation-apps-nfs-mycompany-nfs-provisioner-66fd6b784-lp6dq_d33c2841-0caf-4b29-b087-61e3fbe9a24e External provisioner is provisioning volume for claim "mycompany-cloud/mediation-apps-nfs"
Warning ProvisioningFailed 2m19s (x4 over 5m49s) mediation.apps.nfs.mycompany.nfs.provisioner.io_mediation-apps-nfs-mycompany-nfs-provisioner-66fd6b784-lp6dq_d33c2841-0caf-4b29-b087-61e3fbe9a24e failed to provision volume with StorageClass "mediation-apps-nfs-mycompany-nfs-provisioner": error creating export for volume: error exporting export block
EXPORT
{
Export_Id = 1;
Path = /export/pvc-32c863f3-a086-4cc1-9e19-f809f41a8bec;
Pseudo = /export/pvc-32c863f3-a086-4cc1-9e19-f809f41a8bec;
Access_Type = RW;
Squash = no_root_squash;
SecType = sys;
Filesystem_id = 1.1;
FSAL {
Name = VFS;
}
}
: error calling org.ganesha.nfsd.exportmgr.AddExport: 0 export entries in /export/vfs.conf added because (export create, block validation) errors. Details:
Config File (/export/vfs.conf:54): 1 validation errors in block FSAL
Config File (/export/vfs.conf:54): Errors processing block (FSAL)
Config File (/export/vfs.conf:45): 1 validation errors in block EXPORT
Config File (/export/vfs.conf:45): Errors found in configuration block EXPORT
Normal ExternalProvisioning 6s (x26 over 6m18s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "mediation.apps.nfs.mycompany.nfs.provisioner.io" or manually created by system administrator
Abd this is the error in the nfs pod
I0408 10:20:25.839702 1 main.go:65] Provisioner mediation.apps.nfs.mycompany.nfs.provisioner.io specified
I0408 10:20:25.839822 1 main.go:89] Setting up NFS server!
I0408 10:20:26.019674 1 server.go:149] starting RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0408 10:20:26.019702 1 server.go:160] ending RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0408 10:20:26.023483 1 server.go:134] Running NFS server!
E0408 10:20:31.139572 1 controller.go:908] error syncing claim "32c863f3-a086-4cc1-9e19-f809f41a8bec": failed to provision volume with StorageClass "mediation-apps-nfs-mycompany-nfs-provisioner": error getting NFS server IP for volume: service SERVICE_NAME=mediation-apps-nfs-mycompany-nfs-provisioner is not valid; check that it has for ports map[{111 TCP}:true {111 UDP}:true {662 TCP}:true {662 UDP}:true {875 TCP}:true {875 UDP}:true {2049 TCP}:true {2049 UDP}:true {20048 TCP}:true {20048 UDP}:true {32803 TCP}:true {32803 UDP}:true] exactly one endpoint, this pod's IP POD_IP=10.42.0.151
I0408 10:20:46.146203 1 provision.go:450] using service SERVICE_NAME=mediation-apps-nfs-mycompany-nfs-provisioner cluster IP 10.43.203.168 as NFS server IP
E0408 10:20:46.149486 1 controller.go:908] error syncing claim "32c863f3-a086-4cc1-9e19-f809f41a8bec": failed to provision volume with StorageClass "mediation-apps-nfs-mycompany-nfs-provisioner": error creating export for volume: error exporting export block
EXPORT
{
Export_Id = 1;
Path = /export/pvc-32c863f3-a086-4cc1-9e19-f809f41a8bec;
Pseudo = /export/pvc-32c863f3-a086-4cc1-9e19-f809f41a8bec;
Access_Type = RW;
Squash = no_root_squash;
SecType = sys;
Filesystem_id = 1.1;
FSAL {
Name = VFS;
}
}
: error calling org.ganesha.nfsd.exportmgr.AddExport: 0 export entries in /export/vfs.conf added because (export create, block validation) errors. Details:
Config File (/export/vfs.conf:54): 1 validation errors in block FSAL
Config File (/export/vfs.conf:54): Errors processing block (FSAL)
Config File (/export/vfs.conf:45): 1 validation errors in block EXPORT
Config File (/export/vfs.conf:45): Errors found in configuration block EXPORT
v3.0.0
I'm using the helm chart with the following values:
replicaCount: 3
persistence:
enabled: true
## Persistent Volume Storage Class
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
## set, choosing the default provisioner. (gp2 on AWS, standard on
## GKE, AWS & OpenStack)
##
storageClass: "do-block-storage"
accessMode: ReadWriteOnce
size: 10Gi
## For creating the StorageClass automatically:
storageClass:
create: true
## Set a provisioner name. If unset, a name will be generated.
# provisionerName:
## Set StorageClass as the default StorageClass
## Ignored if storageClass.create is false
defaultClass: false
## Set a StorageClass name
## Ignored if storageClass.create is false
name: nfs
# set to null to prevent expansion
allowVolumeExpansion: true
## StorageClass parameters
parameters: {}
mountOptions:
- vers=3
## ReclaimPolicy field of the class, which can be either Delete or Retain
reclaimPolicy: Delete
When I try to create pvc
it stay in the pending state with the following events:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Provisioning 7s (x2 over 22s) cluster.local/pro-nfs-nfs-server-provisioner_pro-nfs-nfs-server-provisioner-0_9522a528-c992-48c2-bdc3-6dec4370104d External provisioner is provisioning volume for claim "default/test-rwx"
Warning ProvisioningFailed 7s (x2 over 22s) cluster.local/pro-nfs-nfs-server-provisioner_pro-nfs-nfs-server-provisioner-0_9522a528-c992-48c2-bdc3-6dec4370104d failed to provision volume with StorageClass "nfs": error getting NFS server IP for volume: service SERVICE_NAME=pro-nfs-nfs-server-provisioner is not valid; check that it has for ports map[{111 TCP}:true {111 UDP}:true {662 TCP}:true {662 UDP}:true {875 TCP}:true {875 UDP}:true {2049 TCP}:true {2049 UDP}:true {20048 TCP}:true {20048 UDP}:true {32803 TCP}:true {32803 UDP}:true] exactly one endpoint, this pod's IP POD_IP=10.244.1.183
Normal ExternalProvisioning 2s (x3 over 22s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "cluster.local/pro-nfs-nfs-server-provisioner" or manually created by system administrator
If I change the replicaCount
to 1 everything goes fine!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.