k8s中部署并使用Ceph持久化存储

这里使用的rook项目在k8s中部署ceph
https://github.com/rook/rook

1. k8s中部署ceph

1.1 创建operator

1
2
3

[admin@haifly-bj-dev-k8s-master1 ~]$ git clone https://github.com/rook/rook
[admin@haifly-bj-dev-k8s-master1 ~]$ cd ceph/rook/cluster/examples/kubernetes/ceph/
[admin@haifly-bj-dev-k8s-master1 ceph]$ kubectl create -f operator.yaml

1.2 创建cluster

1	[admin@haifly-bj-dev-k8s-master1 ceph]$ kubectl create -f cluster.yaml

1.3 创建dashboard

[admin@haifly-bj-dev-k8s-master1 ceph]$ kubectl create -f dashboard-external-http
从容器rook-ceph-mgr获取密码：
[admin@haifly-bj-dev-k8s-master1 ceph]$ kubectl logs -n rook-ceph rook-ceph-mgr-a-68cb58b456-5wvt2 | grep passw
2019-07-12 12:05:45.305 7fce56298700  0 log_channel(audit) log [DBG] : from='client.94140 172.20.184.188:0/2108025610' entity='client.admin' cmd=[{"username": "admin", "prefix": "dashboard set-login-credentials", "password": "JUnoobgPI1", "target": ["mgr", ""], "format": "json"}]: dispatch

如果没有查到密码，可以toolsbox容器中重新创建一个
[admin@haifly-bj-dev-k8s-master1 ~]$ kubectl exec -it -n rook-ceph rook-ceph-tools-bffbf4d8f-jtd5p  /bin/bash
[root@haifly-bj-dev-k8s-node2 /]# ceph dashboard set-login-credentials admin JUnoobgPI1

1.4 创建mon-service

[admin@haifly-bj-dev-k8s-master1 ceph]$ cat mon-service.yaml 
---
apiVersion: v1
kind: Service
metadata:
  name: rook-ceph-mon
  namespace: rook-ceph
spec:
  type: NodePort 
  ports:
  - port: 6790
    nodePort: 6790
  selector:
    app: rook-ceph-mon

1.5 ceph监控

1	[admin@haifly-bj-dev-k8s-master1 ceph]$ kubectl create -f monitoring/

1.6 ceph tool工具

1	[admin@haifly-bj-dev-k8s-master1 ceph]$ kubectl create -f toolbox.yaml

2.使用ceph rdb

在测试中发现rdb模式下一个pv下只能创建一次pvc，如果把该pv下的pvc删除再次创建，该pv会一直处于Released状态pvc处理Pending状态无法完成绑定。

2.1 静态pvc

2.2.1 创建pool

1	[admin@haifly-bj-dev-k8s-master1 ceph]$ kubectl create -f pool.yaml

2.2.2 创建Secret

[admin@haifly-bj-dev-k8s-master1 ceph]$ cat ceph-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
data:
  key: QVFBdlZnWmJoQ3NhRWhBQU95SlZGWWJadVJESnBRR3BKRERhc3c9PQo=

 #此处的key是tool容器通过cat /etc/ceph/admin.secret|base64获取
[admin@haifly-bj-dev-k8s-master1 ceph]$ kubectl create -f ceph-secret.yml

2.2.3 创建pv

cat << EOF >> test.pv.yml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-pv
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce 
  rbd:
    monitors:
      - 192.168.1.11:6789
      - 192.168.1.12:6789
      - 192.168.1.13:6789
    pool: replicapool
    image: replicapool
    user: admin
    secretRef:
      name: ceph-secret
    fsType: xfs
    readOnly: false
  persistentVolumeReclaimPolicy: Recycle
EOF

kubectl create -f test.pv.yml

2.2.4 创建pvc

cat << EOF >> test.pvc.yml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeName: test-pv
  resources:
    requests:
      storage: 2Gi
EOF

kubectl create -f test.pvc.yml

2.2.5 创建deployment

cat << EOF >> test.deploy.yml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
      - name: demo
        image: mritd/demo
        ports:
        - containerPort: 80
        volumeMounts:
          - mountPath: "/data"
            name: data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: test-pvc
EOF

kubectl create -f test.deploy.yml

2.2 动态pvc（StoragaClass 方式）

2.2.1 创建pool 已有请忽略

1	[admin@haifly-bj-dev-k8s-master1 ceph]$ kubectl create -f pool.yaml

2.2.2 创建StorageClass

cat << EOF >> test.storageclass.yml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: test-storageclass
provisioner: kubernetes.io/rbd
parameters:
  monitors: 192.168.1.11:6789,192.168.1.12:6789,192.168.1.13:6789
  # Ceph 客户端用户 ID(非 k8s 的)
  adminId: admin
  adminSecretName: ceph-secret
  adminSecretNamespace: kube-system
  pool: data
  userId: admin
  userSecretName: ceph-secret
EOF

[admin@haifly-bj-dev-k8s-master1 ceph]$ kubectl create -f storageclass.yaml

2.2.3 创建pvc

[admin@haifly-bj-dev-k8s-master1 ceph]$ cat grafana-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: grafana-pvc
  namespace: monitoring
  annotations:
    volume.beta.kubernetes.io/storage-class: rook-ceph-block
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

2.2.4 创建 Deployment

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: demo-sc
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: demo-sc
    spec:
      containers:
      - name: demo-sc
        image: mritd/demo
        ports:
        - containerPort: 80
        volumeMounts:
          - mountPath: "/data"
            name: data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: grafana-pvc

3.使用ceph fs

Kubernetes跨节点挂载CephFS
注意：所有k8s节点上必须具备2个硬性条件（1.本地存在/etc/ceph/admin.secret文件，文件内容为上文提到的key值，并且如果使用secret形式挂载时，需要将其改为base64格式；2.每个k8s node上必须安装ceph-common cli tools用于调用；）
同时挂载形式也有两种，一种是通过读文件密码信息，一种是通过secret.yaml获取

在K8s中，至少可以通过两种方式挂载CephFS，一种是通过Pod直接挂载；另外一种则是通过pv和pvc挂载。我们分别来看。

3.1 创建ceph fs（pool）

ceph当前正式版功能中仅支持一个fs，对多个fs的支持仅存在于实验feature中

1	[admin@haifly-bj-dev-k8s-master1 ceph]$ kubectl create filesystem.yaml

3.2 Pod直接挂载CephFS

cat ceph-pod2-with-secret.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ceph-pod2-with-secret
spec:
  containers:
 - name: ceph-ubuntu2
    image: ubuntu:16.04
    command: ["tail", "-f", "/var/log/bootstrap.log"]
    volumeMounts:
    - name: ceph-vol2
      mountPath: /mnt/cephfs/data
      readOnly: false
  volumes:
 - name: ceph-vol2
    cephfs:
      monitors:
      - 10.1.0.13:6789
      user: admin
      secretFile: "/etc/ceph/admin.secret"
      readOnly: false

3.3 通过PV和PVC挂载CephFS

3.3.1 创建secret

cat ceph-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
data:
  key: QVFBdlZnWmJoQ3NhRWhBQU95SlZGWWJadVJESnBRR3BKRERhc3c9PQo=

 #此处的key是通过cat /etc/ceph/admin.secret|base64获取

3.3.2 创建pv,pvc

cat ceph-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: foo-pv
spec:
  capacity:
    storage: 512Mi
  accessModes:
    - ReadWriteMany
  cephfs:
    monitors:
      - 10.1.0.13:6789
    path: /
    user: admin
    secretRef:
      name: ceph-secret
    readOnly: false
  persistentVolumeReclaimPolicy: Recycle

$ cat ceph-pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: foo-claim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 512Mi

3.3.3 创建pod

cat ceph-pod2.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ceph-pod2
spec:
  containers:
 - name: ceph-ubuntu2
    image: ubuntu:14.04
    command: ["tail", "-f", "/var/log/bootstrap.log"]
    volumeMounts:
    - name: ceph-vol2
      mountPath: /mnt/cephfs/data
      readOnly: false
  volumes:
 - name: ceph-vol2
    persistentVolumeClaim:
      claimName: foo-claim

3.4 现在使用的ceph fs挂载方式：

3.4.1pv,pvc

apiVersion: v1
kind: PersistentVolume
metadata:
  name: cephfs-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  flexVolume:
    driver: ceph.rook.io/rook
    fsType: ceph
    options:
        fsName: ceph-fs
        path: /grafana
        clusterNamespace: rook-ceph
        size: 1000m
        volumegroup: grafana
  #cephfs:
  #  monitors:
  #    - 192.168.10.177:6790
  #    - 192.168.10.178:6790
  #    - 192.168.9.151:6790
  #  path: /
  #  user: admin
  #  readOnly: false
  #  secretRef:
  #    name: ceph-secret
  persistentVolumeReclaimPolicy: Recycle
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: cephfs-pvc
  namespace: monitoring
spec:
  accessModes:
    - ReadWriteMany
  volumeName: cephfs-pv
  resources:
    requests:
      storage: 1Gi

注意：pv和pvc设置的容量大小并不能限制cephfs某一个目录的大小，也就是说容器可以向cephfs目录写入超出1G的文件。

此外：这里的/grafana目录是提前挂载了/目录后在其它容器中创建的/grafana目录前chmod 777权限，grafana是使用非root运行，不然对目录读写会受限制。

3.4.2 pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
  namespace: dubbo
spec:
  selector:
    matchLabels:
      app: test
  replicas: 
  revisionHistoryLimit: 5
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - name: test
        image: harbor.feiersmart.local/develop/accservice:latest
        imagePullPolicy: Always
        volumeMounts:
        - name: test
          mountPath: /work/admin/test
        ports:
        - containerPort: 20881
          protocol: TCP
        livenessProbe:
          tcpSocket:
            port: 20881
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 3
        env:
        - name: JAVA_OPTS
          value: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap"
        resources:
          requests:
            memory: "400Mi"
          limits:
            memory: "512Mi"
      imagePullSecrets:
      - name: harbor-dev
      volumes:
      - name: test
        persistentVolumeClaim:
          claimName: test-pvc
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - accservice
            topologyKey: "kubernetes.io/hostname"
---
apiVersion: v1
kind: Service
metadata:
  name: test
  namespace: dubbo
spec:
  ports:
  - port: 20881
  selector:
    app: test