极客天成有个很厉害的scaleflash网络文件系统，充分利用rdma的无损网络特性，并进一步发扬光大。研发出来了可以顶替EMC盘阵的存储系统，可以在其上跑Oracle等数据库系统，真的是国货之光了。
帮它们改写了一个csi的存储插件，记录一下，基于yandex的s3 csi而来，这样既可以跑到底层上提供文件块设备，也可以上升到类似NFS或者S3的层次提供文件系统，能满足大多数需求。那源代码就绝对不提供了，只说一下过程：
CSI PLUGIN的使用方法：
一、导入镜像
csi存储插件的image
1cr.yandex/crp9ftr22d26age3hulg/csi-s3:0.42.66
文件：csi.tar
将文件放到所有worknode上，导入本地镜像库:
1ctr --address /run/k3s/containerd/containerd.sock -n k8s.io images import /root/csi.tar
2
3crictl -r unix:///run/k3s/containerd/containerd.sock images
看到有 cr.yandex/crp9ftr22d26age3hulg/csi-s3:0.42.66 既是导入成功
二、安装controller和nodeserver
安装driver
文件：driver.yaml
1kubectl apply -f driver.yaml
安装controller
文件：controller.yaml
1kubectl apply -f controller.yaml
安装nodeserver
文件：nodeserver.yaml
解释一下，controller是一个statefulset，整个集群运行一个即可；而nodeservershi则是一个daemonset，每个worknode都会运行一个副本
上面 worknode 是3个，所以有3个副本
三、scaleflash 块设备的使用
块设备最小单位是G，所以如果需求了100M，那也是1G。
首先必须定义一个storageclass，以后使用这一个storageclass就可以了：
1cat << EOF >scaleflash-storageclass.yaml
2---
3kind: StorageClass
4apiVersion: storage.k8s.io/v1
5metadata:
6 name: scaleflash
7provisioner: ru.yandex.s3.csi
8parameters:
9 mounter: scaleflash
10 clstID: "nvmatrix_101"
11 fsType: "xfs"
12EOF
13
14kubectl apply -f scaleflash-storageclass.yaml
再定义pvc
1cat << EOF > csi-pvc.yaml
2apiVersion: v1
3kind: PersistentVolumeClaim
4metadata:
5 name: csi-pvc
6 namespace: default
7spec:
8 accessModes:
9 - ReadWriteOnce
10 resources:
11 requests:
12 storage: 100M
13 storageClassName: scaleflash
14EOF
15
16kubectl apply -f csi-pvc.yaml
定义Pod使用这个pvc
1cat << EOF > nginx-csi-pvc.yaml
2---
3apiVersion: v1
4kind: Pod
5metadata:
6 name: nginx-pod-s3
7spec:
8 containers:
9 - name: nginx-pod-s3
10 image: docker.io/library/nginx:latest
11 imagePullPolicy: IfNotPresent
12 volumeMounts:
13 - mountPath: "/usr/share/nginx/html"
14 name: dynamic-storage
15 volumes:
16 - name: dynamic-storage
17 persistentVolumeClaim:
18 claimName: csi-pvc
19EOF
20
21kubectl apply -f nginx-csi-pvc.yaml
进入pod，看到lun已经被mount上了，就ok了
反向删除掉Pod和pvc资源：
1kubectl delete -f nginx-csi-pvc.yaml
2kubectl delete -f csi-pvc.yaml
验证statefulset:
1cat << EOF > stateful.yaml
2apiVersion: apps/v1
3kind: StatefulSet
4metadata:
5 name: statefulset-smb
6 namespace: default
7 labels:
8 app: nginx
9spec:
10 serviceName: statefulset-smb
11 replicas: 3
12 template:
13 metadata:
14 labels:
15 app: nginx
16 spec:
17 containers:
18 - name: statefulset-smb
19 image: docker.io/library/nginx:latest
20 imagePullPolicy: IfNotPresent
21 command:
22 - "/bin/bash"
23 - "-c"
24 - set -euo pipefail; while true; do echo $(date) >> /mnt/data/outfile; sleep 1; done
25 volumeMounts:
26 - name: persistent-storage
27 mountPath: /mnt/data
28 readOnly: false
29 updateStrategy:
30 type: RollingUpdate
31 selector:
32 matchLabels:
33 app: nginx
34 volumeClaimTemplates:
35 - metadata:
36 name: persistent-storage
37 namespace: default
38 spec:
39 storageClassName: scaleflash
40 accessModes:
41 - ReadWriteOnce
42 resources:
43 requests:
44 storage: 1Gi
45EOF
46
47kubectl apply -f stateful.yaml
可以放缩副本数，然后观察 /mnt/data/outfile 是否带有刚启动时候的时间戳来确定卷是否是保留的。
四、nvfile的使用（类似NFS）
注意点：rootPath 必须是存在/mnt下，因为只有/mnt被挂进了controller，挂/进去是不行的，所以其它的都无法识别出来
卷容量大小跟NFS一样没有意义，写了也不起任何作用。
定义storageclass
1cat << EOF > storageclass01.yaml
2---
3kind: StorageClass
4apiVersion: storage.k8s.io/v1
5metadata:
6 name: nvfile
7provisioner: ru.yandex.s3.csi
8parameters:
9 mounter: nvfile
10 rootPath: "/mnt/nvfile"
11 modePerm: "0777"
12EOF
13
14kubectl apply -f storageclass01.yaml
注意上面，没有定义subPath，所以新目录都会被建立在/mnt/nvfile下。
如果要定义在某个子目录（比如prod）下， 可以再定义一个storageClass，到时候引用这个新storageclass即可
1---
2kind: StorageClass
3apiVersion: storage.k8s.io/v1
4metadata:
5 name: nvfile-prod
6provisioner: ru.yandex.s3.csi
7parameters:
8 mounter: nvfile
9 rootPath: "/mnt/nvfile"
10 subPath: "/prod"
11 modePerm: "0777"
定义pvc
1cat << EOF > nvfile-pvc.yaml
2---
3apiVersion: v1
4kind: PersistentVolumeClaim
5metadata:
6 name: nvfile-pvc
7 namespace: default
8spec:
9 accessModes:
10 - ReadWriteOnce
11 resources:
12 requests:
13 storage: 100M
14 storageClassName: nvfile
15EOF
16
17kubectl apply -f nvfile-pvc.yaml
定义一个Pod，使用上面的pvc
1cat << EOF > nginx-nvfile.yaml
2---
3apiVersion: v1
4kind: Pod
5metadata:
6 name: nginx-nvfile
7spec:
8 containers:
9 - name: nginx-nvfile
10 image: docker.io/library/nginx:latest
11 imagePullPolicy: IfNotPresent
12 volumeMounts:
13 - mountPath: "/usr/share/nginx/html"
14 name: dynamic-storage
15 volumes:
16 - name: dynamic-storage
17 persistentVolumeClaim:
18 claimName: nvfile-pvc
19EOF
20
21kubectl apply -f nginx-nvfile.yaml
进入容器，看到nvfile_nodev的mount点就是成功
statefulset的验证：
1cat << EOF > stateful-nvfile.yaml
2---
3apiVersion: apps/v1
4kind: StatefulSet
5metadata:
6 name: statefulset-nvfile-ng
7 namespace: default
8 labels:
9 app: nvfile-nginx
10spec:
11 serviceName: statefulset-nvfie-ng
12 replicas: 3
13 template:
14 metadata:
15 labels:
16 app: nvfile-nginx
17 spec:
18 containers:
19 - name: statefulset-nvfile-ng
20 image: docker.io/library/nginx:latest
21 imagePullPolicy: IfNotPresent
22 command:
23 - "/bin/bash"
24 - "-c"
25 - set -euo pipefail; while true; do echo $(date) >> /mnt/smb/outfile; sleep 1; done
26 volumeMounts:
27 - name: pvc
28 mountPath: /mnt/smb
29 readOnly: false
30 updateStrategy:
31 type: RollingUpdate
32 selector:
33 matchLabels:
34 app: nvfile-nginx
35 volumeClaimTemplates:
36 - metadata:
37 name: pvc
38 spec:
39 storageClassName: nvfile
40 accessModes: ["ReadWriteOnce"]
41 resources:
42 requests:
43 storage: 1Gi
44EOF
45
46kubectl apply -f stateful-nvfile.yaml
同样进行伸缩，查看mount点上的文件/mnt/smb/outfile，是否带有刚启动时候的时间戳来判断是否为原始卷
五、S3的使用
S3的话就没有任何约束，只要能跑S3协议，虚机也可以用。
首先建立个minio的S3来模拟，因为9000端口被占，所以用8000端口
http://192.168.66.101:8000
username: abcdefg
password: abcdefg
进入后，gen一对key，赋予S3的所有权限，确保可以建立新bucket
1{
2 "Version": "2012-10-17",
3 "Statement": [
4 {
5 "Effect": "Allow",
6 "Action": [
7 "s3:*"
8 ],
9 "Resource": [
10 "arn:aws:s3:::*"
11 ]
12 }
13 ]
14}
key:
1 accessKeyID: lj7mL2gAFgRCykTaaabbb
2 secretAccessKey: 0YklYzUmcxYcPjZKtmvHRN3cMQrUaCraaaabbbb
然后建立secret
1cat << EOF > secret-s3.yaml
2---
3apiVersion: v1
4kind: Secret
5metadata:
6 name: csi-s3-secret
7 # Namespace depends on the configuration in the storageclass.yaml
8 namespace: kube-system
9stringData:
10 accessKeyID: lj7mL2gAFgRCyaaaabbbb
11 secretAccessKey: 0YklYzUmcxYcPjZKtmvHRN3cMQrUaCraaaabbbb
12 # For AWS set it to "https://s3.<region>.amazonaws.com", for example https://s3.eu-central-1.amazonaws.com
13 endpoint: http://192.168.66.101:8000
14 # For AWS set it to AWS region
15 #region: ""
16EOF
17
18kubectl apply -f secret-s3.yaml
再建立storageclass
1cat << EOF > s3-storageclass.yaml
2---
3kind: StorageClass
4apiVersion: storage.k8s.io/v1
5metadata:
6 name: csi-s3
7provisioner: ru.yandex.s3.csi
8parameters:
9 mounter: geesefs
10 # you can set mount options here, for example limit memory cache size (recommended)
11 options: "--memory-limit 1000 --dir-mode 0777 --file-mode 0666"
12 # to use an existing bucket, specify it here:
13 #bucket: some-existing-bucket
14 csi.storage.k8s.io/provisioner-secret-name: csi-s3-secret
15 csi.storage.k8s.io/provisioner-secret-namespace: kube-system
16 csi.storage.k8s.io/controller-publish-secret-name: csi-s3-secret
17 csi.storage.k8s.io/controller-publish-secret-namespace: kube-system
18 csi.storage.k8s.io/node-stage-secret-name: csi-s3-secret
19 csi.storage.k8s.io/node-stage-secret-namespace: kube-system
20 csi.storage.k8s.io/node-publish-secret-name: csi-s3-secret
21 csi.storage.k8s.io/node-publish-secret-namespace: kube-system
22EOF
23
24kubectl apply -f s3-storageclass.yaml
建立pvc
1cat << EOF > pvc-s3.yaml
2---
3apiVersion: v1
4kind: PersistentVolumeClaim
5metadata:
6 name: csi-s3-pvc
7 namespace: default
8spec:
9 accessModes:
10 - ReadWriteMany
11 resources:
12 requests:
13 storage: 5Gi
14 storageClassName: csi-s3
15EOF
16
17kubectl apply -f csi-s3.yaml
建立pod
1cat << EOF > nginx-s3.yaml
2---
3apiVersion: v1
4kind: Pod
5metadata:
6 name: csi-s3-test-nginx
7 namespace: default
8spec:
9 containers:
10 - name: csi-s3-test-nginx
11 image: docker.io/library/nginx:latest
12 imagePullPolicy: IfNotPresent
13 volumeMounts:
14 - mountPath: /usr/share/nginx/html/s3
15 name: webroot
16 volumes:
17 - name: webroot
18 persistentVolumeClaim:
19 claimName: csi-s3-pvc
20 readOnly: false
21EOF
22
23kubectl apply -f nginx-s3.yaml
进入容器，看到一个pvc的卷即可
去minio的界面，看到这个新卷对应的桶
statefulset也一样。
s3更具体的可以参看：https://github.com/yandex-cloud/k8s-csi-s3/
