NVMeoF CSI

About

This repo contains a CSI (Container Storage Interface) plugin for Kubernetes, designed to provision and manage NVMe over Fabrics (NVMeoF) volumes.

It dynamically provisions NVMe namespaces on the storage backend and exposes them to Kubernetes workloads using NVMeoF.

Please see NvmeOF for detailed introduction.

Supported platforms

This plugin conforms to CSI Spec v1.11.0. It is currently developed and tested only on Kubernetes.

This plugin supports x86_64 and Arm64 architectures.

Project status

Status: Beta

Prerequisites

NVMeoF-CSI is currently developed and tested with Go 1.24, Docker 28.2 and Kubernetes 1.25.0 on CentOS Stream 9.

Before deploying the NVMeoF CSI driver, ensure that your storage backend has an NVMe over Fabrics subsystem installed (inside Ceph cluster) and reachable from your Kubernetes nodes. Follow this guide to install and configure NVMeoF:

NVMe-oF Setup Guide (nvme-cli)

Setup

Build

Compile the Go-based CSI driver binary and build/push its container image:

# Build the CSI driver binary
$ make all

# Build and push the Docker image to your registry
$ docker build -t quay.io/your-org/nvmeof-csi:latest .
$ docker push quay.io/your-org/nvmeof-csi:latest

Usage

Example deployment files can be found in deploy/kubernetes directory.

File	Purpose
`storageclass.yaml`	StorageClass for the NVMeoF CSI driver
`controller.yaml`	StatefulSet for CSI Controller service
`node.yaml`	DaemonSet for CSI Node service
`controller-rbac.yaml`	RBAC for CSI Controller
`node-rbac.yaml`	RBAC for CSI Node
`config-map.yaml`	NVMeoF backend configuration
`secret.yaml`	Access credentials
`driver.yaml`	CSIDriver object
`testpod.yaml`	Example Pod + PVC to verify functionality

Deploy

Start your Kubernetes cluster (e.g., via Minikube):

cd scripts
sudo ./minikube.sh up
sudo ln -sf /var/lib/minikube/binaries/v1.25.0/kubectl /usr/local/bin/kubectl

Deploy CSI services:

cd deploy/kubernetes
./deploy.sh
kubectl get pods

Run a test workload:

kubectl apply -f testpod.yaml

# Verify PersistentVolumes and Claims
kubectl get pv
kubectl get pvc

# Verify the test pod
kubectl get pods

# Check the NVMe-oF volume mount inside the pod
kubectl exec spdkcsi-test -- mount | grep nvme

Teardown

Clean up the test resources and cluster:

# Delete test workload
kubectl delete -f deploy/kubernetes/testpod.yaml

# Tear down CSI services
cd deploy/kubernetes
./deploy.sh teardown

# Clean up Kubernetes cluster
cd scripts
sudo ./minikube.sh clean

Architecture & Flow

Below are two detailed ASCII diagrams showing:

Controller Side – how kube-controller-manager calls your CSI Controller, which in turn talks to the NVMe-oF Gateway.
Node Side – how kubelet interacts with your CSI Node to perform attach/mount operations.

1. Controller & NVMe-oF Gateway

+--------------------------------------------------+
| Kubernetes Control Plane                         |
|  +--------------------------------------------+  |
|  | CSI Controller Deployment (nvmeofcsi)     |  |
|  |  • CreateVolume()                         |  |
|  |  • DeleteVolume()                         |  |
|  |  • ControllerPublishVolume()              |  |
|  +--------------------------------------------+  |
+--------------------------------------------------+
                   │
         CSI gRPC  │                         NVMe-oF Gateway SDK/gRPC
  (drivername=csi.nvmeof.io)           (e.g. management IP:5500)
                   ▼
+--------------------------------------------------+
| NVMe-oF Gateway gRPC Server                      |
|  • NamespaceAdd(volumeName, size, parameters)    |
|  • NamespaceDel(namespaceID)                     |
|  • ListSubsystems(), ListNamespaces()            |
+--------------------------------------------------+

Flow:

CreateVolume → CSI Controller
Controller issues NamespaceAdd → NVMe-oF Gateway
Gateway provisions an NVMe namespace and returns NQN, TRADDR, TRSVCID, transport.
CSI Controller records these in VolumeContext.
On ControllerPublishVolume, Controller may export additional attachment details.

2. Node DaemonSet & Kubelet

+-----------------------------+        Unix Socket                        +-----------------------------+
| kubelet (Pod volume plugin) | <--------------------------------------> | CSI Node Server (_out/nvmeofcsi) |
+-----------------------------+    /var/lib/kubelet/plugins/ \            +-----------------------------+
                                            csi.nvmeof.io/     
    ▲                    │
    │ NodeGetInfo        │
    │ NodeStageVolume    │
    │  • run `nvme discover` + `nvme connect` using VolumeContext  
    │ NodePublishVolume  │
    │  • bind-mount device into Pod’s filesystem
    │ NodeUnstageVolume  │
    │ NodeUnpublishVolume│
    │                    ▼
+------------------------------------------------------------------+
|    NVMe-oF Target(s)                                             |
|    • Subsystems, Namespaces exported over TCP transports         |
+------------------------------------------------------------------+

Flow:

kubelet invokes NodeStageVolume → CSI Node
Node runs nvme discover, nvme connect with – nqn (subsystem name) – traddr (target address) – trsvcid (port) – transport, etc.
On success, device node (e.g. /dev/nvmeXnY) appears on host.
NodePublishVolume → bind-mounts that block device into the Pod’s mount path.
Tear-down via NodeUnpublishVolume / NodeUnstageVolume.

Flow Diagram:

+-------------------------------+                
|  Kubernetes Control Plane     |                
|  (Provisioner, Scheduler)     |                
+---------------+---------------+                
                | PVC created                       
                v                                  
     +----------------------------+                
     |  CSI Controller Plugin     |   <- Deployment 
     | (CreateVolume)             |                
     +----------------------------+                
                |                                  
                | gRPC call: CreateVolume()        
                v                                  
     +-----------------------------+               
     | NVMe-oF Gateway (gRPC)      |               
     | • NamespaceAdd              |               
     +-----------------------------+               
                |                                  
                v                                  
     +-----------------------------+               
     | Volume metadata returned     |              
     | (NQN, NSID, size, etc.)      |              
     +-----------------------------+               
                |                                  
                v                                  
     +-----------------------------+               
     | CSI Controller → Responds   |               
     | to K8s with Volume object   |               
     +-----------------------------+               
                |                                  
                v                                  
+--------------------------------------------+    
| Kubernetes Scheduler assigns Pod to Node   |    
+--------------------------------------------+    
                |                                  
                v                                  
+-------------------------------+                  
| kubelet (on assigned Node)    |                  
| detects volume requirement    |                  
+---------------+---------------+                  
                |                                  
                v                                  
     +---------------------------+                 
     |  CSI Node Plugin          |  <- DaemonSet    
     | (NodeStageVolume)         |                 
     +---------------------------+                 
                |                                  
                | Runs: `nvme discover + connect`  
                v                                  
     +-----------------------------+               
     |  NVMe-oF Target Service     |               
     |  (Subsystems, Namespaces)   |               
     +-----------------------------+               
                |                                  
                v                                  
     +---------------------------+                 
     | NodePublishVolume         |                 
     | • Bind mount device       |                 
     +---------------------------+                 
                |                                  
                v                                  
     +---------------------------+                 
     | Container sees /dev/nvmeX |                 
     +---------------------------+                 

==================== CLEANUP =====================

User deletes Pod + PVC
                |
                v
+---------------------------+
| CSI Node Plugin           |
| NodeUnpublishVolume       |
| NodeUnstageVolume         |
| • Disconnect NVMe device  |
+---------------------------+
                |
                v
+---------------------------+
| CSI Controller Plugin     |
| DeleteVolume              |
| • Calls NamespaceDelete   |
+---------------------------+
                |
                v
+---------------------------+
| NVMe-oF Gateway           |
| Removes NS from subsystem |
+---------------------------+

Communication and Contribution

Please join SPDK community for communication and contribution.

Project progress is tracked in Trello board.

gadididi/nvmeof-csi

README