- A+
所属分类：linux技术
前文我们聊到了k8s上crd资源的使用和相关说明，回顾请参考：https://www.cnblogs.com/qiuhom-1874/p/14267400.html；今天我们来了解下k8s的第二种扩展机制自定义apiserver，以及apiservice资源的相关话题；
在开始聊自定义apiserver前，我们先来了解下k8s原生的apiserver；其实apiserver就是一个https服务器，我们可以使用kubectl工具通过https协议请求apiserver创建资源，删除资源，查看资源等等操作；每个请求都对应着RESTful api中的请求方法，对应资源就是http协议中的url路径；比如我们要创建一个pod，其kubectl请求apiserver 使用post方法将资源定义提交给apiserver；pod资源就是对应群组中的某个版本下某个名称空间下的某个pod资源；
apiserver资源组织逻辑
提示：客户端访问apiserver，对应资源类似上图中的组织方式；比如访问default名称空间下某个pod，其路径就为/apis/core/v1/namespace/default/pod/mypod；对应resource包含名称空间以及对应资源类型；
k8s原生apiserver组成
k8s原生apiserver主要有两个组件组成，第一个组件aggregator，其功能类似web代理服务器，第二个组件就是真正的apiserver；其工作逻辑是，用户请求首先送达给aggregator，由aggregator根据用户请求的资源，将对应请求路由至apiserver；简单讲aggregator这个组件主要作用就是用来路由用户请求；默认情况aggregator会把所有请求都路由至原生的apiserver上进行响应；如果我们需要自定义apiserver，就需要在默认的aggregator上使用apiservice资源将自定义apiserver注册到原生的apiserver上，让其用户请求能够被路由至自定义apiserver进行响应；如下图
提示：apiserver是k8s的唯一访问入口，默认客户端的所有操作都是发送给apiserver进行响应，我们自定义的apiserver要想能够被客户端访问，就必须通过内建apiserver中的aggregator组件中的路由信息，把对应路径的访问路由至对应apiserver进行访问；对应aggregator中的路由信息，由k8s内建apiservice资源定义；简单讲apiservice资源就是用来定义原生apiserver中aggregator组件上的路由信息；该路由就是将某某端点的访问路由至对应apiserver；
查看原生apiserver中的群组/版本信息
[root@master01 ~]# kubectl api-versions admissionregistration.k8s.io/v1 admissionregistration.k8s.io/v1beta1 apiextensions.k8s.io/v1 apiextensions.k8s.io/v1beta1 apiregistration.k8s.io/v1 apiregistration.k8s.io/v1beta1 apps/v1 authentication.k8s.io/v1 authentication.k8s.io/v1beta1 authorization.k8s.io/v1 authorization.k8s.io/v1beta1 autoscaling/v1 autoscaling/v2beta1 autoscaling/v2beta2 batch/v1 batch/v1beta1 certificates.k8s.io/v1 certificates.k8s.io/v1beta1 coordination.k8s.io/v1 coordination.k8s.io/v1beta1 crd.projectcalico.org/v1 discovery.k8s.io/v1beta1 events.k8s.io/v1 events.k8s.io/v1beta1 extensions/v1beta1 flowcontrol.apiserver.k8s.io/v1beta1 mongodb.com/v1 networking.k8s.io/v1 networking.k8s.io/v1beta1 node.k8s.io/v1 node.k8s.io/v1beta1 policy/v1beta1 rbac.authorization.k8s.io/v1 rbac.authorization.k8s.io/v1beta1 scheduling.k8s.io/v1 scheduling.k8s.io/v1beta1 stable.example.com/v1 storage.k8s.io/v1 storage.k8s.io/v1beta1 v1 [root@master01 ~]#
提示：只有上面列出的群组版本才能够被客户端访问，即客户端只能访问上述列表中群组版本中的资源，没有出现群组版本是一定不会被客户端访问到；
apiservice资源的使用
示例：创建apiservice资源
[root@master01 ~]# cat apiservice-demo.yaml apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: name: v2beta1.auth.ilinux.io spec: insecureSkipTLSVerify: true group: auth.ilinux.io groupPriorityMinimum: 1000 versionPriority: 15 service: name: auth-api namespace: default version: v2beta1 [root@master01 ~]#
提示：apiservice资源属于apiregistration.k8s.io/v1群组，其类型为APIService，其中spec.insecureSkipTLSVerify字段用来描述是否忽略安全验证，即不验证https证书；true表示不验证，false表示要验证；group字段用来描述对应自定义apiserver的群组；groupPriorityMinimum字段用来描述对应群组的优先级；versionPriority字段用来描述对应群组版本的优先级；service字段是用来描述把对应群组版本的请求路由至某个service；该service就是对应自定义apiserver关联的service；version字段用来描述对应apiserver属于对应群组中的某个版本；上述资源清单表示在aggregator上注册auth.ilinux.io/v2beta1这个端点，该端点对应的后端apiserver的service是default名称空间下的auth-api service；即客户端访问auth.ilinux.io/v2beta1下的资源都会被路由至default名称空间下的auth-api service进行响应；
应用清单
[root@master01 ~]# kubectl apply -f apiservice-demo.yaml apiservice.apiregistration.k8s.io/v2beta1.auth.ilinux.io created [root@master01 ~]# kubectl get apiservice |grep auth.ilinux.io v2beta1.auth.ilinux.io default/auth-api False (ServiceNotFound) 16s [root@master01 ~]# kubectl api-versions |grep auth.ilinux.io auth.ilinux.io/v2beta1 [root@master01 ~]#
提示：可以看到应用清单以后，对应的端点信息就出现在api-versions中；
上述清单只是用来说明对应apiservice资源的使用；其实应用上述清单创建apiservice资源没有实质的作用，其原因是我们对应名称空间下并没有对应的服务，也没有对应自定义apiserver；所以通常自定义apiserver，我们会用到apiservice资源来把自定义apiserver整合进原生apiserver中；接下来我们部署一个真正意义上的自定义apiserver
部署metrics-server
metrics-server是用来扩展k8s的第三方apiserver，其主要作用是收集pod或node上的cpu，内存，磁盘等指标数据，并提供一个api接口供kubectl top命令访问；默认情况kubectl top 命令是没法正常使用，其原因是默认apiserver上没有对应的接口提供收集pod或node的cpu，内存，磁盘等核心指标数据；kubectl top命令主要用来显示pod/node资源的cpu，内存，磁盘的占用比例；该命令能够正常使用必须依赖Metrics API；
默认没有部署metrics server使用kubectl top pod/node查看pod或节点的cpu，内存占用比例
[root@master01 ~]# kubectl top Display Resource (CPU/Memory/Storage) usage. The top command allows you to see the resource consumption for nodes or pods. This command requires Metrics Server to be correctly configured and working on the server. Available Commands: node Display Resource (CPU/Memory/Storage) usage of nodes pod Display Resource (CPU/Memory/Storage) usage of pods Usage: kubectl top [flags] [options] Use "kubectl <command> --help" for more information about a given command. Use "kubectl options" for a list of global command-line options (applies to all commands). [root@master01 ~]# kubectl top pod error: Metrics API not available [root@master01 ~]# kubectl top node error: Metrics API not available [root@master01 ~]#
提示：默认没有部署metrics server，使用kubectl top pod/node命令，它会告诉我们没有可用的metrics api；
部署metrics server
下载部署清单
[root@master01 ~]# mkdir metrics-server [root@master01 ~]# cd metrics-server [root@master01 metrics-server]# wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.0/components.yaml --2021-01-14 23:54:30-- https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.0/components.yaml Resolving github.com (github.com)... 52.74.223.119 Connecting to github.com (github.com)|52.74.223.119|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/92132038/c700f080-1f7e-11eb-9e30-864a63f442f4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210114T155432Z&X-Amz-Expires=300&X-Amz-Signature=fc5a6f41ca50ec22e87074a778d2cb35e716ae6c3231afad17dfaf8a02203e35&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=92132038&response-content-disposition=attachment%3B%20filename%3Dcomponents.yaml&response-content-type=application%2Foctet-stream [following] --2021-01-14 23:54:32-- https://github-production-release-asset-2e65be.s3.amazonaws.com/92132038/c700f080-1f7e-11eb-9e30-864a63f442f4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210114T155432Z&X-Amz-Expires=300&X-Amz-Signature=fc5a6f41ca50ec22e87074a778d2cb35e716ae6c3231afad17dfaf8a02203e35&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=92132038&response-content-disposition=attachment%3B%20filename%3Dcomponents.yaml&response-content-type=application%2Foctet-stream Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.39.44 Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.39.44|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 3962 (3.9K) [application/octet-stream] Saving to: ‘components.yaml’ 100%[===========================================================================================>] 3,962 11.0KB/s in 0.4s 2021-01-14 23:54:35 (11.0 KB/s) - ‘components.yaml’ saved [3962/3962] [root@master01 metrics-server]# ls components.yaml [root@master01 metrics-server]#
修改部署清单内容
[root@master01 metrics-server]# cat components.yaml apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: metrics-server name: metrics-server namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: k8s-app: metrics-server rbac.authorization.k8s.io/aggregate-to-admin: "true" rbac.authorization.k8s.io/aggregate-to-edit: "true" rbac.authorization.k8s.io/aggregate-to-view: "true" name: system:aggregated-metrics-reader rules: - apiGroups: - metrics.k8s.io resources: - pods - nodes verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: k8s-app: metrics-server name: system:metrics-server rules: - apiGroups: - "" resources: - pods - nodes - nodes/stats - namespaces - configmaps verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: k8s-app: metrics-server name: metrics-server-auth-reader namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: extension-apiserver-authentication-reader subjects: - kind: ServiceAccount name: metrics-server namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: k8s-app: metrics-server name: metrics-server:system:auth-delegator roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegator subjects: - kind: ServiceAccount name: metrics-server namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: k8s-app: metrics-server name: system:metrics-server roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:metrics-server subjects: - kind: ServiceAccount name: metrics-server namespace: kube-system --- apiVersion: v1 kind: Service metadata: labels: k8s-app: metrics-server name: metrics-server namespace: kube-system spec: ports: - name: https port: 443 protocol: TCP targetPort: https selector: k8s-app: metrics-server --- apiVersion: apps/v1 kind: Deployment metadata: labels: k8s-app: metrics-server name: metrics-server namespace: kube-system spec: selector: matchLabels: k8s-app: metrics-server strategy: rollingUpdate: maxUnavailable: 0 template: metadata: labels: k8s-app: metrics-server spec: containers: - args: - --cert-dir=/tmp - --secure-port=4443 - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname - --kubelet-use-node-status-port - --kubelet-insecure-tls image: k8s.gcr.io/metrics-server/metrics-server:v0.4.0 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /livez port: https scheme: HTTPS periodSeconds: 10 name: metrics-server ports: - containerPort: 4443 name: https protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /readyz port: https scheme: HTTPS periodSeconds: 10 securityContext: readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 1000 volumeMounts: - mountPath: /tmp name: tmp-dir nodeSelector: kubernetes.io/os: linux priorityClassName: system-cluster-critical serviceAccountName: metrics-server volumes: - emptyDir: {} name: tmp-dir --- apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: labels: k8s-app: metrics-server name: v1beta1.metrics.k8s.io spec: group: metrics.k8s.io groupPriorityMinimum: 100 insecureSkipTLSVerify: true service: name: metrics-server namespace: kube-system version: v1beta1 versionPriority: 100 [root@master01 metrics-server]#
提示：在deploy中，spec.template.containers.args字段中加上--kubelet-insecure-tls选项，表示不验证客户端证书；上述清单主要用deploy控制器将metrics server运行为一个pod，然后授权metrics-server用户能够对pod/node资源进行只读权限；然后把metrics.k8s.io/v1beta1注册到原生apiserver上，让其客户端访问metrics.k8s.io下的资源能够被路由至metrics-server这个服务上进行响应；
应用资源清单
[root@master01 metrics-server]# kubectl apply -f components.yaml serviceaccount/metrics-server created clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created clusterrole.rbac.authorization.k8s.io/system:metrics-server created rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created service/metrics-server created deployment.apps/metrics-server created apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created [root@master01 metrics-server]#
验证：查看原生apiserver是否有metrics.k8s.io/v1beta1?
[root@master01 metrics-server]# kubectl api-versions|grep metrics metrics.k8s.io/v1beta1 [root@master01 metrics-server]#
提示：可以看到metrics.k8s.io/v1beta1群组已经注册到原生apiserver上；
查看metrics server pod是否运行正常？
[root@master01 metrics-server]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-744cfdf676-kh6rm 1/1 Running 4 5d7h canal-5bt88 2/2 Running 20 11d canal-9ldhl 2/2 Running 22 11d canal-fvts7 2/2 Running 20 11d canal-mwtg4 2/2 Running 23 11d canal-rt8nn 2/2 Running 21 11d coredns-7f89b7bc75-k9gdt 1/1 Running 32 37d coredns-7f89b7bc75-kp855 1/1 Running 31 37d etcd-master01.k8s.org 1/1 Running 36 37d kube-apiserver-master01.k8s.org 1/1 Running 14 13d kube-controller-manager-master01.k8s.org 1/1 Running 43 37d kube-flannel-ds-fnd2w 1/1 Running 5 5d5h kube-flannel-ds-k9l4k 1/1 Running 7 5d5h kube-flannel-ds-s7w2j 1/1 Running 4 5d5h kube-flannel-ds-vm4mr 1/1 Running 6 5d5h kube-flannel-ds-zgq92 1/1 Running 37 37d kube-proxy-74fxn 1/1 Running 10 10d kube-proxy-fbl6c 1/1 Running 8 10d kube-proxy-n82sf 1/1 Running 10 10d kube-proxy-ndww5 1/1 Running 11 10d kube-proxy-v8dhk 1/1 Running 11 10d kube-scheduler-master01.k8s.org 1/1 Running 39 37d metrics-server-58fcfcc9d-drbw2 1/1 Running 0 32s [root@master01 metrics-server]#
提示：可以看到对应pod已经正常运行；
查看pod里的日志是否正常？
[root@master01 metrics-server]# kubectl logs metrics-server-58fcfcc9d-drbw2 -n kube-system I0114 17:52:03.601493 1 serving.go:325] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key) E0114 17:52:04.140587 1 pathrecorder.go:107] registered "/metrics" from goroutine 1 [running]: runtime/debug.Stack(0x1942e80, 0xc00069aed0, 0x1bb58b5) /usr/local/go/src/runtime/debug/stack.go:24 +0x9d k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).trackCallers(0xc00028afc0, 0x1bb58b5, 0x8) /go/pkg/mod/k8s.io/apiserver@v0.19.2/pkg/server/mux/pathrecorder.go:109 +0x86 k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).Handle(0xc00028afc0, 0x1bb58b5, 0x8, 0x1e96f00, 0xc0006d88d0) /go/pkg/mod/k8s.io/apiserver@v0.19.2/pkg/server/mux/pathrecorder.go:173 +0x84 k8s.io/apiserver/pkg/server/routes.MetricsWithReset.Install(0xc00028afc0) /go/pkg/mod/k8s.io/apiserver@v0.19.2/pkg/server/routes/metrics.go:43 +0x5d k8s.io/apiserver/pkg/server.installAPI(0xc00000a1e0, 0xc000589b00) /go/pkg/mod/k8s.io/apiserver@v0.19.2/pkg/server/config.go:711 +0x6c k8s.io/apiserver/pkg/server.completedConfig.New(0xc000589b00, 0x1f099c0, 0xc0001449b0, 0x1bbdb5a, 0xe, 0x1ef29e0, 0x2cef248, 0x0, 0x0, 0x0) /go/pkg/mod/k8s.io/apiserver@v0.19.2/pkg/server/config.go:657 +0xb45 sigs.k8s.io/metrics-server/pkg/server.Config.Complete(0xc000589b00, 0xc000599440, 0xc000599b00, 0xdf8475800, 0xc92a69c00, 0x0, 0x0, 0xdf8475800) /go/src/sigs.k8s.io/metrics-server/pkg/server/config.go:52 +0x312 sigs.k8s.io/metrics-server/cmd/metrics-server/app.runCommand(0xc00001c6e0, 0xc000114600, 0x0, 0x0) /go/src/sigs.k8s.io/metrics-server/cmd/metrics-server/app/start.go:66 +0x157 sigs.k8s.io/metrics-server/cmd/metrics-server/app.NewMetricsServerCommand.func1(0xc0000d9340, 0xc0005a4cd0, 0x0, 0x5, 0x0, 0x0) /go/src/sigs.k8s.io/metrics-server/cmd/metrics-server/app/start.go:37 +0x33 github.com/spf13/cobra.(*Command).execute(0xc0000d9340, 0xc00013a130, 0x5, 0x5, 0xc0000d9340, 0xc00013a130) /go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x453 github.com/spf13/cobra.(*Command).ExecuteC(0xc0000d9340, 0xc00013a180, 0x0, 0x0) /go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x349 github.com/spf13/cobra.(*Command).Execute(...) /go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887 main.main() /go/src/sigs.k8s.io/metrics-server/cmd/metrics-server/metrics-server.go:38 +0xae I0114 17:52:04.266492 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0114 17:52:04.267021 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController I0114 17:52:04.266641 1 secure_serving.go:197] Serving securely on [::]:4443 I0114 17:52:04.266670 1 tlsconfig.go:240] Starting DynamicServingCertificateController I0114 17:52:04.266682 1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key I0114 17:52:04.266688 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0114 17:52:04.267120 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0114 17:52:04.266692 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0114 17:52:04.267301 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0114 17:52:04.367448 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0114 17:52:04.367472 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0114 17:52:04.367462 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController [root@master01 metrics-server]#
提示：只要metrics server pod没有出现错误日志，或者无法注册等信息，就表示pod里的容器运行正常；
验证：使用kubectl top 命令查看pod的cpu ，内存占比，看看对应命令是否可以正常执行？
[root@master01 metrics-server]# kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% master01.k8s.org 235m 11% 1216Mi 70% node01.k8s.org 140m 3% 747Mi 20% node02.k8s.org 120m 3% 625Mi 17% node03.k8s.org 133m 3% 594Mi 16% node04.k8s.org 125m 3% 700Mi 19% [root@master01 metrics-server]# kubectl top pods -n kube-system NAME CPU(cores) MEMORY(bytes) calico-kube-controllers-744cfdf676-kh6rm 2m 23Mi canal-5bt88 50m 118Mi canal-9ldhl 22m 86Mi canal-fvts7 49m 106Mi canal-mwtg4 57m 113Mi canal-rt8nn 56m 113Mi coredns-7f89b7bc75-k9gdt 3m 12Mi coredns-7f89b7bc75-kp855 3m 15Mi etcd-master01.k8s.org 25m 72Mi kube-apiserver-master01.k8s.org 99m 410Mi kube-controller-manager-master01.k8s.org 14m 88Mi kube-flannel-ds-fnd2w 3m 45Mi kube-flannel-ds-k9l4k 3m 27Mi kube-flannel-ds-s7w2j 4m 46Mi kube-flannel-ds-vm4mr 3m 45Mi kube-flannel-ds-zgq92 2m 19Mi kube-proxy-74fxn 1m 27Mi kube-proxy-fbl6c 1m 23Mi kube-proxy-n82sf 1m 25Mi kube-proxy-ndww5 1m 25Mi kube-proxy-v8dhk 2m 23Mi kube-scheduler-master01.k8s.org 3m 33Mi metrics-server-58fcfcc9d-drbw2 6m 23Mi [root@master01 metrics-server]#
提示：可以看到kubectl top命令可以正常执行，说明metrics server 部署成功没有问题；
以上就是使用apiservice资源结合自定义apiserver扩展k8s功能的示例，简单总结apiservice资源的主要作用就是在aggregator上创建对应的路由信息，该路由信息的主要作用是将对应端点访问路由至自定义apiserver所对应的service进行响应；
