- A+
所属分类：linux技术
前文我们了解了k8s节点污点和pod的对节点污点容忍度相关话题，回顾请参考：https://www.cnblogs.com/qiuhom-1874/p/14255486.html；今天我们来聊一下扩展k8s相关话题；
k8s上创建资源对象的过程
我们知道在k8s上，资源是有类型的，不同类型的资源，其定义方式和使用的字段各不相同；用户创建一个资源，实际上就是把k8s抽象的资源做实例化，即把k8s抽象出来的资源，通过资源清单赋值，创建出来的对象就是我们实例化对应类型资源的结果；用户创建一个资源，首先会把请求发送给apiserver，通过apiserver的认证、授权、准入控制以后，对应创建资源的定义就存放在etcd中，控制器通过watch机制监视apiserver上的资源变动，通过对应资源变动事件触发对应类型资源的控制器将对应资源创建出来，并通过控制器内部的和解循环监控着对应资源状态是否和用户定义的期望状态一样，如果发现不一样，内部和解循环就会被触发，对应控制器会向apiserver发起创建资源的请求，将对应资源重建，让对应资源的状态始终满足用户期望的状态；从上述的过程来看，用户创建一个资源分两个步骤，第一步是将对应请求发送给apiserver，通过apiserver把对应资源定义的信息存放在etcd中；第二个步骤是对应资源类型的控制器通过apiserver从etcd中读取对应资源的定义，将其创建出来；对于etcd来说，它本是就一个kv数据库，可以存储任意类型的kv数据，但在在k8s上，apiserver将不同类型的资源定义抽象成不同的资源，使得用户创建对应资源，必须是满足对应类型资源定义的规范，然后将规范的定义存放在etcd中；简单讲apiserver就是把用户存入etcd中的数据做了一层抽象，使得用户不能够随意将任意数据存储到etcd中，存入etcd中的数据必须是满足对应apiserver接口定义的规范；这就好比我们在使用mysql数据库时，必须遵守对应库中的对应表的定义；
在k8s上创建自定义资源类型
在k8s上，资源的类型有很多，比如pod，service，PersistentVolume，PersistentVolumeClaim等等，这些都是一些基础的资源类型；我们要创建某种资源，直接使用对应的资源类型，实例化一个对象即可；假如我们要在k8s上创建一个集群，我们是不是可以直接使用某种类型的资源，实例化一个集群对象呢？理论上是可以的，但是前提是对应k8s上有对应类型的资源；有对应类型的资源，用户就可以把对应创建资源的定义存放在etcd中；除了有对应类型的资源，我们还需要有对应的控制器将对应资源创建出来；这样一来对于不同集群或应用来说，其组织方式和逻辑都不同，使用的资源类型和控制器也有所不同；用户要想实例化更高级的资源来，就必须自己手动实现定义其资源类型，将对应资源类型实例化为对象；除此之外必要时还需要自己实现对应资源的控制器；简单讲用户想要实现更高级的资源类型，就必须扩展现有k8s的资源类型和控制器；
在k8s上扩展资源类型的方式有三种，第一种是crd，crd是k8s内建的资源类型，该类型资源主要用来创建用户自定义资源类型的资源；即通过crd资源，可以将用户自定义资源类型转换为k8s上资源类型；第二种是自定义apiserver；这种方式要比第一种方式要复杂一点，需要用户手动开发程序实现对应功能的apiserver，让其用户创建自定义类型资源能够通过自定义apiserver实现；第三种方式就是修改现有k8sapiserver，让其支持对应用户自定义资源类型；
自定义控制器
自定义资源类型我们可以使用crd资源实现，也可以使用自定义apiserver或修改原有apiserver代码实现，但是只有资源类型是不能够让对应自定义类型资源实例化为一个自定义资源对象，只有自定义资源类型，用户创建对应资源类型的资源对象时，只能把对应资源类型的定义信息写入到etcd中，它不能真正的跑起来，要想真正的跑起来，我们还需要一个自定义控制器，专门负责监听对应的资源类型的资源变化，将对应资源实例化为对应k8s上的资源对象；当然不是所有的自定义类型的资源都需要自定义控制器，如果对应自定义类型资源调用了底层基础控制器来管控对应自定义资源，那么对应自定义类型资源就不需要使用自定义控制器；我们知道控制器是k8s上的一个重要组件，它的工作逻辑是注册监听在apiserver上对应类型的资源变动，如果对应资源状态不满足用户期望状态，它就会根据内部的和解循环来请求apiserver将对应类型资源的定义发送给它，然后根据资源定义来重建对应的资源，让其状态始终和用户期望的状态保持一致；自定义控制器也是同样的逻辑，使用自定义控制器的目的也是让对应自定义类型资源能够被自定义控制器监听，一旦对应资源发生变动，它能够将其在k8s上创建出来，并一直保持和用户期望的状态吻合；自定义控制器和自定义资源类型可以分开实现，也可以合并在一起实现，即自定义控制器程序能够自动创建crd资源，让其对应自定义类型资源能够被k8s识别并将其创建出来；具体是分开实现还是合并在一起实现，取决开发自定义控制器程序员；
crd资源定义帮助
[root@master01 ~]# kubectl explain crd KIND: CustomResourceDefinition VERSION: apiextensions.k8s.io/v1 DESCRIPTION: CustomResourceDefinition represents a resource that should be exposed on the API server. Its name MUST be in the format <.spec.name>.<.spec.group>. FIELDS: apiVersion <string> APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources kind <string> Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds metadata <Object> spec <Object> -required- spec describes how the user wants the resources to appear status <Object> status indicates the actual state of the CustomResourceDefinition [root@master01 ~]#
提示：crd资源是k8s上的标准资源之一，它的定义主要有apiVersion，kind，metadata，spec和status；其中kind类型为CustomResourceDefinition，apiVersion是apiextensions.k8s.io/v1；这两个是固定格式；spec字段是用来定义对应指定资源类型资源的相关属性；
crd.spec字段说明
[root@master01 ~]# kubectl explain crd.spec KIND: CustomResourceDefinition VERSION: apiextensions.k8s.io/v1 RESOURCE: spec <Object> DESCRIPTION: spec describes how the user wants the resources to appear CustomResourceDefinitionSpec describes how a user wants their resource to appear FIELDS: conversion <Object> conversion defines conversion settings for the CRD. group <string> -required- group is the API group of the defined custom resource. The custom resources are served under `/apis/<group>/...`. Must match the name of the CustomResourceDefinition (in the form `<names.plural>.<group>`). names <Object> -required- names specify the resource and kind names for the custom resource. preserveUnknownFields <boolean> preserveUnknownFields indicates that object fields which are not specified in the OpenAPI schema should be preserved when persisting to storage. apiVersion, kind, metadata and known fields inside metadata are always preserved. This field is deprecated in favor of setting `x-preserve-unknown-fields` to true in `spec.versions[*].schema.openAPIV3Schema`. See https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#pruning-versus-preserving-unknown-fields for details. scope <string> -required- scope indicates whether the defined custom resource is cluster- or namespace-scoped. Allowed values are `Cluster` and `Namespaced`. versions <[]Object> -required- versions is the list of all API versions of the defined custom resource. Version names are used to compute the order in which served versions are listed in API discovery. If the version string is "kube-like", it will sort above non "kube-like" version strings, which are ordered lexicographically. "Kube-like" versions start with a "v", then are followed by a number (the major version), then optionally the string "alpha" or "beta" and another number (the minor version). These are sorted first by GA > beta > alpha (where GA is a version with no suffix such as beta or alpha), and then by comparing major version, then minor version. An example sorted list of versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1, v11alpha2, foo1, foo10. [root@master01 ~]#
提示：crd.spec中group字段使用来描述对应自定义类型资源的群组名称，其值为字符串；names字段是用来描述自定义类型资源对应类型，名称等等，其值为一个对象；scope字段用来定义对应自定义资源是那个级别的资源；该字段的值只能为Cluster或Namespaced；versions字段是用来指定对应自定义资源的版本，以及对应类型资源的属性字段等信息，该字段为一个列表对象；
示例：定义一个自定义类型资源
[root@master01 ~]# cat crontab-crd.yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: # 名字必需与下面的 spec 字段匹配，并且格式为 '<名称的复数形式>.<组名>' name: crontabs.stable.example.com spec: # 组名称，用于 REST API: /apis/<组>/<版本> group: stable.example.com # 列举此 CustomResourceDefinition 所支持的版本 versions: - name: v1 # 每个版本都可以通过 served 标志来独立启用或禁止 served: true # 其中一个且只有一个版本必需被标记为存储版本 storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: cronSpec: type: string image: type: string replicas: type: integer # 可以是 Namespaced 或 Cluster scope: Namespaced names: # 名称的复数形式，用于 URL：/apis/<组>/<版本>/<名称的复数形式> plural: crontabs # 名称的单数形式，作为命令行使用时和显示时的别名 singular: crontab # kind 通常是单数形式的驼峰编码（CamelCased）形式。你的资源清单会使用这一形式。 kind: CronTab # shortNames 允许你在命令行使用较短的字符串来匹配资源 shortNames: - ct [root@master01 ~]#
应用资源清单前使用kubectl get crontab
[root@master01 ~]# kubectl get crontab error: the server doesn't have a resource type "crontab" [root@master01 ~]#
提示：在没有应用资源清单前使用kubectl get crontab，它会提示我们所没有类型为crontab的资源
应用资源清单
[root@master01 ~]# kubectl apply -f crontab-crd.yaml customresourcedefinition.apiextensions.k8s.io/crontabs.stable.example.com created [root@master01 ~]# kubectl get crontab No resources found in default namespace. [root@master01 ~]#
提示：应用资源清单后，再次使用kubectl get crontab就没有报错了，只是提示在default名称空间没有对应类型资源；
查看crd
[root@master01 ~]# kubectl get crd NAME CREATED AT bgpconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z bgppeers.crd.projectcalico.org 2021-01-03T15:49:21Z blockaffinities.crd.projectcalico.org 2021-01-03T15:49:21Z clusterinformations.crd.projectcalico.org 2021-01-03T15:49:21Z crontabs.stable.example.com 2021-01-12T12:39:00Z felixconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z globalnetworkpolicies.crd.projectcalico.org 2021-01-03T15:49:21Z globalnetworksets.crd.projectcalico.org 2021-01-03T15:49:21Z hostendpoints.crd.projectcalico.org 2021-01-03T15:49:21Z ipamblocks.crd.projectcalico.org 2021-01-03T15:49:21Z ipamconfigs.crd.projectcalico.org 2021-01-03T15:49:21Z ipamhandles.crd.projectcalico.org 2021-01-03T15:49:21Z ippools.crd.projectcalico.org 2021-01-03T15:49:21Z kubecontrollersconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z networkpolicies.crd.projectcalico.org 2021-01-03T15:49:21Z networksets.crd.projectcalico.org 2021-01-03T15:49:22Z [root@master01 ~]# kubectl get crd/crontabs.stable.example.com NAME CREATED AT crontabs.stable.example.com 2021-01-12T12:39:00Z [root@master01 ~]#
查看详情
[root@master01 ~]# kubectl get crd/crontabs.stable.example.com NAME CREATED AT crontabs.stable.example.com 2021-01-12T12:39:00Z [root@master01 ~]# kubectl describe crd/crontabs.stable.example.com Name: crontabs.stable.example.com Namespace: Labels: <none> Annotations: <none> API Version: apiextensions.k8s.io/v1 Kind: CustomResourceDefinition Metadata: Creation Timestamp: 2021-01-12T12:39:00Z Generation: 1 Managed Fields: API Version: apiextensions.k8s.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: f:acceptedNames: f:kind: f:listKind: f:plural: f:shortNames: f:singular: f:conditions: Manager: kube-apiserver Operation: Update Time: 2021-01-12T12:39:00Z API Version: apiextensions.k8s.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:spec: f:conversion: .: f:strategy: f:group: f:names: f:kind: f:listKind: f:plural: f:shortNames: f:singular: f:scope: f:versions: f:status: f:storedVersions: Manager: kubectl-client-side-apply Operation: Update Time: 2021-01-12T12:39:00Z Resource Version: 805506 UID: b92a90f4-c953-4876-a496-030c9ba023fd Spec: Conversion: Strategy: None Group: stable.example.com Names: Kind: CronTab List Kind: CronTabList Plural: crontabs Short Names: ct Singular: crontab Scope: Namespaced Versions: Name: v1 Schema: openAPIV3Schema: Properties: Spec: Properties: Cron Spec: Type: string Image: Type: string Replicas: Type: integer Type: object Type: object Served: true Storage: true Status: Accepted Names: Kind: CronTab List Kind: CronTabList Plural: crontabs Short Names: ct Singular: crontab Conditions: Last Transition Time: 2021-01-12T12:39:00Z Message: no conflicts found Reason: NoConflicts Status: True Type: NamesAccepted Last Transition Time: 2021-01-12T12:39:00Z Message: the initial names have been accepted Reason: InitialNamesAccepted Status: True Type: Established Stored Versions: v1 Events: <none> [root@master01 ~]#
使用自定义资源类型crontab创建资源
[root@master01 ~]# cat my-crontab.yaml apiVersion: "stable.example.com/v1" kind: CronTab metadata: name: my-new-cron-object spec: cronSpec: "* * * * */5" image: my-awesome-cron-image [root@master01 ~]#
提示：上述资源清单表示创建一个类型为Crontab的资源，该资源的群组版本为stable.example.com/v1;
应用资源清单
[root@master01 ~]# kubectl apply -f my-crontab.yaml crontab.stable.example.com/my-new-cron-object created [root@master01 ~]# kubectl get ct NAME AGE my-new-cron-object 5s [root@master01 ~]# kubectl describe ct/my-new-cron-object Name: my-new-cron-object Namespace: default Labels: <none> Annotations: <none> API Version: stable.example.com/v1 Kind: CronTab Metadata: Creation Timestamp: 2021-01-12T12:45:29Z Generation: 1 Managed Fields: API Version: stable.example.com/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:spec: .: f:cronSpec: f:image: Manager: kubectl-client-side-apply Operation: Update Time: 2021-01-12T12:45:29Z Resource Version: 806182 UID: 31a88a3d-fa99-42b8-80f6-3e4559efdc40 Spec: Cron Spec: * * * * */5 Image: my-awesome-cron-image Events: <none> [root@master01 ~]#
提示：可以看到对应类型资源已经创建成功；以上示例只是单纯的crd的使用示例，没有任何实质的作用；
部署自定义控制器
示例：部署mongodb-aperator
1、克隆项目
[root@master01 ~]# git clone https://github.com/mongodb/mongodb-kubernetes-operator.git Cloning into 'mongodb-kubernetes-operator'... remote: Enumerating objects: 95, done. remote: Counting objects: 100% (95/95), done. remote: Compressing objects: 100% (74/74), done. remote: Total 4506 (delta 30), reused 60 (delta 15), pack-reused 4411 Receiving objects: 100% (4506/4506), 18.04 MiB | 183.00 KiB/s, done. Resolving deltas: 100% (2621/2621), done. [root@master01 ~]#
2、创建名称空间mongodb，并进入到mongodb-kubernetes-operator目录应用crd资源，创建自定义资源类型
[root@master01 mongodb-kubernetes-operator]# kubectl create ns mongodb namespace/mongodb created [root@master01 mongodb-kubernetes-operator]# kubectl get ns NAME STATUS AGE default Active 35d ingress-nginx Active 22d kube-node-lease Active 35d kube-public Active 35d kube-system Active 35d kubernetes-dashboard Active 11d mongodb Active 4s [root@master01 mongodb-kubernetes-operator]# ls agent build deploy docs go.sum pkg release.json scripts testdata version APACHE2 cmd dev_notes go.mod LICENSE.md README.md requirements.txt test tools.go [root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_mongodb_crd.yaml -n mongodb Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition customresourcedefinition.apiextensions.k8s.io/mongodb.mongodb.com created [root@master01 mongodb-kubernetes-operator]#
验证：查看mongodb类型资源是否已经创建成功？
[root@master01 mongodb-kubernetes-operator]# kubectl get crd NAME CREATED AT bgpconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z bgppeers.crd.projectcalico.org 2021-01-03T15:49:21Z blockaffinities.crd.projectcalico.org 2021-01-03T15:49:21Z clusterinformations.crd.projectcalico.org 2021-01-03T15:49:21Z crontabs.stable.example.com 2021-01-12T12:39:00Z felixconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z globalnetworkpolicies.crd.projectcalico.org 2021-01-03T15:49:21Z globalnetworksets.crd.projectcalico.org 2021-01-03T15:49:21Z hostendpoints.crd.projectcalico.org 2021-01-03T15:49:21Z ipamblocks.crd.projectcalico.org 2021-01-03T15:49:21Z ipamconfigs.crd.projectcalico.org 2021-01-03T15:49:21Z ipamhandles.crd.projectcalico.org 2021-01-03T15:49:21Z ippools.crd.projectcalico.org 2021-01-03T15:49:21Z kubecontrollersconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z mongodb.mongodb.com 2021-01-13T06:38:22Z networkpolicies.crd.projectcalico.org 2021-01-03T15:49:21Z networksets.crd.projectcalico.org 2021-01-03T15:49:22Z [root@master01 mongodb-kubernetes-operator]# kubectl get crd/mongodb.mongodb.com NAME CREATED AT mongodb.mongodb.com 2021-01-13T06:38:22Z [root@master01 mongodb-kubernetes-operator]#
3、安装operator
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/operator/ -n mongodb deployment.apps/mongodb-kubernetes-operator created role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created serviceaccount/mongodb-kubernetes-operator created [root@master01 mongodb-kubernetes-operator]#
提示：mongodb-kubernetes-operator这个项目是将自定义控制器和自定义资源类型分开实现的；其operator只负责创建和监听对应资源类型的变化，在资源有变化时，实例化为对应资源对象，并保持对应资源对象状态吻合用户期望状态；上述四个清单中主要是创建了一个sa账户，并对对应的sa用户授权；
operator.yaml内容
[root@master01 mongodb-kubernetes-operator]# cat deploy/operator/operator.yaml apiVersion: apps/v1 kind: Deployment metadata: name: mongodb-kubernetes-operator spec: replicas: 1 selector: matchLabels: name: mongodb-kubernetes-operator template: metadata: labels: name: mongodb-kubernetes-operator spec: serviceAccountName: mongodb-kubernetes-operator containers: - name: mongodb-kubernetes-operator image: quay.io/mongodb/mongodb-kubernetes-operator:0.3.0 command: - mongodb-kubernetes-operator imagePullPolicy: Always env: - name: WATCH_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: OPERATOR_NAME value: "mongodb-kubernetes-operator" - name: AGENT_IMAGE # The MongoDB Agent the operator will deploy to manage MongoDB deployments value: quay.io/mongodb/mongodb-agent:10.19.0.6562-1 - name: VERSION_UPGRADE_HOOK_IMAGE value: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.2 - name: MONGODB_IMAGE value: "library/mongo" - name: MONGODB_REPO_URL value: "registry.hub.docker.com" [root@master01 mongodb-kubernetes-operator]#
提示：上述资源主要用deploy控制器运行对应自定义控制器为一个pod；
验证：查看operator是否正常运行
[root@master01 mongodb-kubernetes-operator]# kubectl get pods -n mongodb NAME READY STATUS RESTARTS AGE mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 26s [root@master01 mongodb-kubernetes-operator]#
提示：能够看到operator正常运行，就表示operator已经安装成功；
验证：使用自定义资源类型创建一个mongodb 副本集集群
[root@master01 mongodb-kubernetes-operator]# cat deploy/crds/mongodb.com_v1_mongodb_cr.yaml --- apiVersion: mongodb.com/v1 kind: MongoDB metadata: name: example-mongodb spec: members: 3 type: ReplicaSet version: "4.2.6" security: authentication: modes: ["SCRAM"] users: - name: my-user db: admin passwordSecretRef: # a reference to the secret that will be used to generate the user's password name: my-user-password roles: - name: clusterAdmin db: admin - name: userAdminAnyDatabase db: admin scramCredentialsSecretName: my-scram # the user credentials will be generated from this secret # once the credentials are generated, this secret is no longer required --- apiVersion: v1 kind: Secret metadata: name: my-user-password type: Opaque stringData: password: 58LObjiMpxcjP1sMDW [root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_v1_mongodb_cr.yaml mongodb.mongodb.com/example-mongodb created secret/my-user-password created [root@master01 mongodb-kubernetes-operator]#
应用清单
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_v1_mongodb_cr.yaml -n mongodb mongodb.mongodb.com/example-mongodb created secret/my-user-password created [root@master01 mongodb-kubernetes-operator]# kubectl get pods -n mongodb NAME READY STATUS RESTARTS AGE example-mongodb-0 0/2 Pending 0 9s mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 88s [root@master01 mongodb-kubernetes-operator]#
提示：这里可以看到对应pod处于pending状态；
查看pod详细信息
[root@master01 mongodb-kubernetes-operator]# kubectl describe pod/example-mongodb-0 -n mongodb|grep -A 10 "Events" Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 66s (x2 over 66s) default-scheduler 0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims. [root@master01 mongodb-kubernetes-operator]#
提示：这里提示没有可以用的pvc；
删除mongodb名称空间下pvc
[root@master01 mongodb-kubernetes-operator]# kubectl get pvc -n mongodb NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-volume-example-mongodb-0 Pending 92s [root@master01 mongodb-kubernetes-operator]# kubectl delete pvc --all -n mongodb persistentvolumeclaim "data-volume-example-mongodb-0" deleted [root@master01 mongodb-kubernetes-operator]# kubectl get pvc -n mongodb No resources found in mongodb namespace. [root@master01 mongodb-kubernetes-operator]#
创建pv和pvc
[root@master01 mongodb-kubernetes-operator]# cd [root@master01 ~]# cat pv-demo.yaml apiVersion: v1 kind: PersistentVolume metadata: name: nfs-pv-v1 labels: app: example-mongodb-svc spec: capacity: storage: 1Gi volumeMode: Filesystem accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"] persistentVolumeReclaimPolicy: Retain mountOptions: - hard - nfsvers=4.1 nfs: path: /data/v1 server: 192.168.0.99 --- apiVersion: v1 kind: PersistentVolume metadata: name: nfs-pv-v2 labels: app: example-mongodb-svc spec: capacity: storage: 1Gi volumeMode: Filesystem accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"] persistentVolumeReclaimPolicy: Retain mountOptions: - hard - nfsvers=4.1 nfs: path: /data/v2 server: 192.168.0.99 --- apiVersion: v1 kind: PersistentVolume metadata: name: nfs-pv-v3 labels: app: example-mongodb-svc spec: capacity: storage: 1Gi volumeMode: Filesystem accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"] persistentVolumeReclaimPolicy: Retain mountOptions: - hard - nfsvers=4.1 nfs: path: /data/v3 server: 192.168.0.99 [root@master01 ~]#
应用清单创建pv
[root@master01 ~]# kubectl apply -f pv-demo.yaml persistentvolume/nfs-pv-v1 created persistentvolume/nfs-pv-v2 created persistentvolume/nfs-pv-v3 created [root@master01 ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE nfs-pv-v1 1Gi RWO,ROX,RWX Retain Available 3s nfs-pv-v2 1Gi RWO,ROX,RWX Retain Available 3s nfs-pv-v3 1Gi RWO,ROX,RWX Retain Available 3s [root@master01 ~]#
创建pvc清单
[root@master01 ~]# cat pvc-demo.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-volume-example-mongodb-0 namespace: mongodb spec: accessModes: - ReadWriteMany volumeMode: Filesystem resources: requests: storage: 500Mi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-volume-example-mongodb-1 namespace: mongodb spec: accessModes: - ReadWriteMany volumeMode: Filesystem resources: requests: storage: 500Mi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-volume-example-mongodb-2 namespace: mongodb spec: accessModes: - ReadWriteMany volumeMode: Filesystem resources: requests: storage: 500Mi [root@master01 ~]#
应用清单创建pvc
[root@master01 ~]# kubectl get pvc -n mongodb No resources found in mongodb namespace. [root@master01 ~]# kubectl apply -f pvc-demo.yaml persistentvolumeclaim/data-volume-example-mongodb-0 created persistentvolumeclaim/data-volume-example-mongodb-1 created persistentvolumeclaim/data-volume-example-mongodb-2 created [root@master01 ~]# kubectl get pvc -n mongodb NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-volume-example-mongodb-0 Bound nfs-pv-v1 1Gi RWO,ROX,RWX 6s data-volume-example-mongodb-1 Bound nfs-pv-v2 1Gi RWO,ROX,RWX 6s data-volume-example-mongodb-2 Bound nfs-pv-v3 1Gi RWO,ROX,RWX 6s [root@master01 ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE nfs-pv-v1 1Gi RWO,ROX,RWX Retain Bound mongodb/data-volume-example-mongodb-0 102s nfs-pv-v2 1Gi RWO,ROX,RWX Retain Bound mongodb/data-volume-example-mongodb-1 102s nfs-pv-v3 1Gi RWO,ROX,RWX Retain Bound mongodb/data-volume-example-mongodb-2 102s [root@master01 ~]#
提示：可以看到对应pvc和pv已经绑定好了；
验证：查看mongodb副本集集群是否跑起来了？
[root@master01 ~]# kubectl get pods -n mongodb NAME READY STATUS RESTARTS AGE example-mongodb-0 2/2 Running 0 6m19s example-mongodb-1 0/2 PodInitializing 0 10s mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 7m38s [root@master01 ~]# kubectl get pods -n mongodb -w NAME READY STATUS RESTARTS AGE example-mongodb-0 2/2 Running 0 6m35s example-mongodb-1 1/2 Running 0 26s mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 7m54s example-mongodb-1 2/2 Running 0 43s example-mongodb-2 0/2 Pending 0 0s example-mongodb-2 0/2 Pending 0 0s example-mongodb-2 0/2 Init:0/1 0 0s example-mongodb-2 0/2 Init:0/1 0 1s example-mongodb-2 0/2 Terminating 0 4s example-mongodb-2 0/2 Terminating 0 6s example-mongodb-2 0/2 Terminating 0 20s example-mongodb-2 0/2 Terminating 0 20s example-mongodb-2 0/2 Pending 0 0s example-mongodb-2 0/2 Pending 0 0s example-mongodb-2 0/2 Init:0/1 0 0s example-mongodb-2 0/2 Init:0/1 0 1s example-mongodb-2 0/2 PodInitializing 0 7s example-mongodb-2 1/2 Running 0 14s example-mongodb-2 2/2 Running 0 36s ^C[root@master01 ~]# kubectl get pods -n mongodb NAME READY STATUS RESTARTS AGE example-mongodb-0 2/2 Running 0 8m example-mongodb-1 2/2 Running 0 111s example-mongodb-2 2/2 Running 0 48s mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 9m19s [root@master01 ~]#
提示：可以看到对应的pod已经正常跑起来了；
验证：使用mongo 连接mongodbpod，看看对应副本集集群是否工作正常？
[root@master01 ~]# kubectl get pods -n mongodb -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES example-mongodb-0 2/2 Running 0 9m12s 10.244.4.101 node04.k8s.org <none> <none> example-mongodb-1 2/2 Running 0 3m3s 10.244.2.130 node02.k8s.org <none> <none> example-mongodb-2 2/2 Running 0 2m 10.244.1.130 node01.k8s.org <none> <none> mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 10m 10.244.3.116 node03.k8s.org <none> <none> [root@master01 ~]# mongo 10.244.4.101 MongoDB shell version v4.4.3 connecting to: mongodb://10.244.4.101:27017/test?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("b9d16fe9-6a74-4638-96e6-70aaf3c83bfa") } MongoDB server version: 4.2.6 WARNING: shell and server versions do not match example-mongodb:PRIMARY> show dbs example-mongodb:PRIMARY> db.auth('my-user','58LObjiMpxcjP1sMDW') Error: Authentication failed. 0 example-mongodb:PRIMARY> use admin switched to db admin example-mongodb:PRIMARY> db.auth('my-user','58LObjiMpxcjP1sMDW') 1 example-mongodb:PRIMARY> show dbs admin 0.000GB config 0.000GB local 0.000GB example-mongodb:PRIMARY> db.isMaster() { "hosts" : [ "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017", "example-mongodb-1.example-mongodb-svc.mongodb.svc.cluster.local:27017", "example-mongodb-2.example-mongodb-svc.mongodb.svc.cluster.local:27017" ], "setName" : "example-mongodb", "setVersion" : 1, "ismaster" : true, "secondary" : false, "primary" : "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017", "me" : "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017", "electionId" : ObjectId("7fffffff0000000000000003"), "lastWrite" : { "opTime" : { "ts" : Timestamp(1610520741, 1), "t" : NumberLong(3) }, "lastWriteDate" : ISODate("2021-01-13T06:52:21Z"), "majorityOpTime" : { "ts" : Timestamp(1610520741, 1), "t" : NumberLong(3) }, "majorityWriteDate" : ISODate("2021-01-13T06:52:21Z") }, "maxBsonObjectSize" : 16777216, "maxMessageSizeBytes" : 48000000, "maxWriteBatchSize" : 100000, "localTime" : ISODate("2021-01-13T06:52:27.873Z"), "logicalSessionTimeoutMinutes" : 30, "connectionId" : 153, "minWireVersion" : 0, "maxWireVersion" : 8, "readOnly" : false, "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1610520741, 1), "signature" : { "hash" : BinData(0,"EcWzL7O9Ue9kmm6cQ4FumkcIP6g="), "keyId" : NumberLong("6917119940596072451") } }, "operationTime" : Timestamp(1610520741, 1) } example-mongodb:PRIMARY>
提示：可以看到三个mongodb pod是副本集关系；其中example-mongodb-0是主节点，其他两个是从节点；
最后还是说一下，我在做上面的实验中，虽然mongodb operator已经正常工作，但是用mongo这个客户端工具连接到主节点没法写数据，提示没有权限；但对应的用户在对应库是有读写权限的；在admin库下创建用户能提示用户添加成功，但是过几秒查询用户信息，发现用户不存在，不知道什么原因，有知道的朋友麻烦告知下（博主邮箱：linux-1874@qq.com），博主将感激不尽。。
