在上一节中，Heketi的部署方式还不能用于生产环境，因为Heketi Pod的数据并没有持久化，容易导致heketi的数据丢失，Heketi的数据保存在/var/lib/heketi/heketi.db文件中，因此需要把此目录挂载到GlusterFS分布式存储中。
按照上一节的步骤，执行heketi-cli topology load --json=topology-sample.json
$ echo $HEKETI_CLI_SERVERhttp://10.254.49.43:8080$ heketi-cli topology load --json=topology-sample.jsonCreating cluster ... ID: 5b930ef6081fd22e895c25a3dfb0c516 ???Allowing file volumes on cluster. ???Allowing block volumes on cluster. ???Creating node 10.30.1.15 ... ID: b120572be40db6c1d979c3903876430b ???????Adding device /dev/sdb ... OK ???Creating node 10.30.1.16 ... ID: 7ce13ffc5eabe64a3791e93233fd3c1a ???????Adding device /dev/sdb ... OK ???Creating node 10.30.1.17 ... ID: f9abdc2e5d4cfa17c035a97f984a1a3b ???????Adding device /dev/sdb ... OK紧接着执行：将生成heketi-storage.json文件。
$ heketi-cli setup-openshift-heketi-storageSaving heketi-storage.json执行完后，查看Pod deploy-heketi日志信息，看看做了哪些事：
#只截取了部分日志，基本操作就是进入到各个glusterfs Pod创建brick目录及创建一个副本为3的Replicate volume, volume名为heketidbstorage[kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir -p /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6Result: [kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.15 Pod: glusterfs-c4859 Command: mkdir -p /var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4Result: [kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: mkdir -p /var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.15 Pod: glusterfs-c4859 Command: mkdir /var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: mkdir /var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brickResult: [cmdexec] INFO 2018/07/09 07:07:26 Creating volume heketidbstorage replica 3[kubeexec] DEBUG 2018/07/09 07:07:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: gluster --mode=script volume create heketidbstorage replica 3 10.30.1.16:/var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick 10.30.1.17:/var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick 10.30.1.15:/var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brickResult: volume create: heketidbstorage: success: please start the volume to access data进入任意GlusterFS Pod查看卷信息：
$ kubectl exec glusterfs-25cm8 -it bash[root@ubuntu16 /]# gluster volume infoVolume Name: heketidbstorageType: ReplicateVolume ID: c8da2a4a-3066-4708-a59d-201d22decd92Status: StartedSnapshot Count: 0Number of Bricks: 1 x 3 = 3Transport-type: tcpBricks:Brick1: 10.30.1.16:/var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brickBrick2: 10.30.1.17:/var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brickBrick3: 10.30.1.15:/var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brickOptions Reconfigured:transport.address-family: inetnfs.disable: on[root@ubuntu16 /]# 分析下heketi-storage.json：
#将创建如下资源信息：Endpoints： ???name：heketi-storage-endpoints ????????10.30.1.16:1 10.30.1.15:1 10.30.1.17:1Service： ???????name: heketi-storage-endpoints ?job： ???name: heketi-storage-copy-job ??images: heketi/heketi:dev ??声明了 volume：heketi-storage ?????????"volumes": [ ???????????????????????????????{ ???????????????????????????????????"name": "heketi-storage", ???????????????????????????????????"glusterfs": { ???????????????????????????????????????"endpoints": "heketi-storage-endpoints", ???????????????????????????????????????"path": "heketidbstorage" ???????????????????????????????????} ???????????????????????????????}, ????挂载到 /heketi： ????????????volumeMounts": [ ???????????????????????????????????????????{ ???????????????????????????????????????????????"name": "heketi-storage", ???????????????????????????????????????????????"mountPath": "/heketi" ???????????????????????????????????????????}, ??启动时执行命令：cp /db/heketi.db /heketi ???#由此可知，此job的作用就是复制heketi中的数据文件到 /heketi，而/heketi目录挂载在了卷heketi-storage中，而heketi-storage volume是前面执行"heketi-cli setup-openshift-heketi-storage"时创建好了的创建之：
$ kubectl create -f heketi-storage.jsonsecret "heketi-storage-secret" createdendpoints "heketi-storage-endpoints" createdservice "heketi-storage-endpoints" createdjob "heketi-storage-copy-job" created当Job执行完后就可以删除它了：
$ kubectl get job NAME ?????????????????????DESIRED ??SUCCESSFUL ??AGEheketi-storage-copy-job ??1 ????????1 ???????????1m等到job完成后，删除bootstrap Heketi实例相关的组件：
#把之前由heketi-bootstrap.json创建的资源删除$ kubectl delete all,service,jobs,deployment,secret --selector="deploy-heketi"deployment "deploy-heketi" deletedjob "heketi-storage-copy-job" deletedpod "deploy-heketi-69bfbd4bbd-q8tsk" deletedservice "deploy-heketi" deletedsecret "heketi-storage-secret" deleted之前创建的名为deploy-heketi的pod,service已经删除了：
$ kubectl get podNAME ?????????????READY ????STATUS ?????????????RESTARTS ??AGEglusterfs-25cm8 ??1/1 ??????Running ????????????1 ?????????1hglusterfs-8qrpt ??1/1 ??????Running ????????????1 ?????????1hglusterfs-c4859 ??1/1 ??????Running ????????????1 ?????????1h$ kubectl get svcNAME ?????????????????????????TYPE ???????CLUSTER-IP ??????EXTERNAL-IP ??PORT(S) ???AGEheketi-storage-endpoints ?????ClusterIP ??10.254.191.233 ??<none> ???????1/TCP ?????4m最后，使用heketi-deployment.json文件重新部署heketi
$ cat heketi-deployment.json { ?"kind": "List", ?"apiVersion": "v1", ?"items": [ ???{ ?????"kind": "Secret", ?????"apiVersion": "v1", ?????"metadata": { ???????"name": "heketi-db-backup", ???????"labels": { ?????????"glusterfs": "heketi-db", ?????????"heketi": "db" ???????} ?????}, ?????"data": { ?????}, ?????"type": "Opaque" ???}, ???{ ?????"kind": "Service", ?????"apiVersion": "v1", ?????"metadata": { ???????"name": "heketi", ???????"labels": { ?????????"glusterfs": "heketi-service", ?????????"deploy-heketi": "support" ???????}, ???????"annotations": { ?????????"description": "Exposes Heketi Service" ???????} ?????}, ?????"spec": { ???????"selector": { ?????????"name": "heketi" ???????}, ???????"ports": [ ?????????{ ???????????"name": "heketi", ???????????"port": 8080, ???????????"targetPort": 8080 ?????????} ???????] ?????} ???}, ???{ ?????"kind": "Deployment", ?????"apiVersion": "extensions/v1beta1", ?????"metadata": { ???????"name": "heketi", ???????"labels": { ?????????"glusterfs": "heketi-deployment" ???????}, ???????"annotations": { ?????????"description": "Defines how to deploy Heketi" ???????} ?????}, ?????"spec": { ???????"replicas": 1, ???????"template": { ?????????"metadata": { ???????????"name": "heketi", ???????????"labels": { ?????????????"name": "heketi", ?????????????"glusterfs": "heketi-pod" ???????????} ?????????}, ?????????"spec": { ???????????"serviceAccountName": "heketi-service-account", ???????????"containers": [ ?????????????{ ???????????????"image": "heketi/heketi:dev", ???????????????"imagePullPolicy": "Always", ???????????????"name": "heketi", ???????????????"env": [ ?????????????????{ ???????????????????"name": "HEKETI_EXECUTOR", ???????????????????"value": "kubernetes" ?????????????????}, ?????????????????{ ???????????????????"name": "HEKETI_DB_PATH", ???????????????????"value": "/var/lib/heketi/heketi.db" ?????????????????}, ?????????????????{ ???????????????????"name": "HEKETI_FSTAB", ???????????????????"value": "/var/lib/heketi/fstab" ?????????????????}, ?????????????????{ ???????????????????"name": "HEKETI_SNAPSHOT_LIMIT", ???????????????????"value": "14" ?????????????????}, ?????????????????{ ???????????????????"name": "HEKETI_KUBE_GLUSTER_DAEMONSET", ???????????????????"value": "y" ?????????????????} ???????????????], ???????????????"ports": [ ?????????????????{ ???????????????????"containerPort": 8080 ?????????????????} ???????????????], ???????????????"volumeMounts": [ ?????????????????{ ???????????????????"mountPath": "/backupdb", ???????????????????"name": "heketi-db-secret" ?????????????????}, ?????????????????{ ???????????????????"name": "db", ???????????????????"mountPath": "/var/lib/heketi" ?????????????????}, ?????????????????{ ???????????????????"name": "config", ???????????????????"mountPath": "/etc/heketi" ?????????????????} ???????????????], ???????????????"readinessProbe": { ?????????????????"timeoutSeconds": 3, ?????????????????"initialDelaySeconds": 3, ?????????????????"httpGet": { ???????????????????"path": "/hello", ???????????????????"port": 8080 ?????????????????} ???????????????}, ???????????????"livenessProbe": { ?????????????????"timeoutSeconds": 3, ?????????????????"initialDelaySeconds": 30, ?????????????????"httpGet": { ???????????????????"path": "/hello", ???????????????????"port": 8080 ?????????????????} ???????????????} ?????????????} ???????????], ???????????"volumes": [ ?????????????{ ???????????????"name": "db", ???????????????"glusterfs": { ?????????????????"endpoints": "heketi-storage-endpoints", ?????????????????"path": "heketidbstorage" ???????????????} ?????????????}, ?????????????{ ???????????????"name": "heketi-db-secret", ???????????????"secret": { ?????????????????"secretName": "heketi-db-backup" ???????????????} ?????????????}, ?????????????{ ???????????????"name": "config", ???????????????"secret": { ?????????????????"secretName": "heketi-config-secret" ???????????????} ?????????????} ???????????] ?????????} ???????} ?????} ???} ?]}heketi-deployment.json 文件 创建了如下资源：
Service: ????name: heketi ????port: 8080Deployment： ???name: heketi ???replicas: 1 ???image: heketi/heketi:dev ???volumeMounts: ???????????name: db ??mountPath: /var/lib/heketi ???volumes: ??????endpoints: heketi-storage-endpoints ??#由heketi-storage.json文件创建 ??????path：heketidbstorage ?#这是gluster volume名，此volume是由"heketi-cli setup-openshift-heketi-storage"自动创建。 ???# 将heketi容器内的/var/lib/heketi 目录挂载到了GlusterFS volume “heketidbstorage”中。部署之：
$ kubectl create -f heketi-deployment.json secret "heketi-db-backup" createdservice "heketi" createddeployment "heketi" created$ kubectl get deployment NAME ?????DESIRED ??CURRENT ??UP-TO-DATE ??AVAILABLE ??AGEheketi ???1 ????????1 ????????1 ???????????1 ??????????45s$ kubectl get svc NAME ?????????????????????????TYPE ???????CLUSTER-IP ??????EXTERNAL-IP ??PORT(S) ???AGEheketi ???????????????????????ClusterIP ??10.254.239.189 ??<none> ???????8080/TCP ??51sheketi-storage-endpoints ?????ClusterIP ??10.254.191.233 ??<none> ???????1/TCP ?????31m验证heketi是否在用在用gluster volume：
$ kubectl get podNAME ?????????????????????READY ????STATUS ?????????????RESTARTS ??AGEglusterfs-25cm8 ??????????1/1 ??????Running ????????????1 ?????????1hglusterfs-8qrpt ??????????1/1 ??????Running ????????????1 ?????????1hglusterfs-c4859 ??????????1/1 ??????Running ????????????1 ?????????1hheketi-7898db85dd-nb6kn ??1/1 ??????Running ????????????0 ?????????1m$ kubectl exec heketi-7898db85dd-nb6kn -it bash[root@heketi-7898db85dd-nb6kn /]# mount |grep heketi10.30.1.15:heketidbstorage on /var/lib/heketi type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)至此，heketi db已正确配置了GlusterFS卷。
kubernetes中部署Heketi和GlusterFS(二)
原文地址：http://blog.51cto.com/newfly/2139393