- A+
所属分类：linux技术
前文我们了解了用Prometheus监控k8s上的节点和pod资源，回顾请参考：https://www.cnblogs.com/qiuhom-1874/p/14287942.html；今天我们来了解下k8s上的HPA资源的使用；
HPA的全称是Horizontal Pod Autoscaler，从字面意思理解它就是水平pod自动伸缩器；简单讲HPA的主要作用是根据指定的指标数据，监控对应的pod控制器，一旦对应pod控制器下的pod的对应指标数据达到我们定义的阀值，即HPA就会被触发，它会根据对应指标数据的值来扩展/缩减对应pod副本数量；扩展和缩减都是有上下限的，当pod数量达到上限，即便指标数据还是超过了我们定义的阀值它也不会再扩展，对于下线默认不指定就是为1；下限和上限都是一样的逻辑，即便一个访问都没有，它会保持最低有多少个pod运行；需注意hpa只能用于监控可扩缩的pod控制器，对DaemonSet类型控制器不支持；
在k8s上HPA有两个版本v1和v2；v1只支持根据cpu这个指标数据来自动扩展/缩减pod数量；V2支持根据自定义指标数量来自动扩展/缩减pod数量；HPA是k8s上的标准资源，我们可以通过命令或资源清单的方式去创建它；
使用命令创建HPA资源的使用语法格式
Usage: kubectl autoscale (-f FILENAME | TYPE NAME | TYPE/NAME) [--min=MINPODS] --max=MAXPODS [--cpu-percent=CPU] [options]
提示：默认不指定hpa的名称，它会同对应的pod控制名称相同；--min选项用于指定对应pod副本最低数量，默认不指定其值为1，--max用于指定pod最大副本数量；--cpu-percent选项用于指定对应pod的cpu资源的占用比例，该占用比例是同对应pod的资源限制做比较；
示例：使用命令创建v1版本的hpa资源
使用deploy控制器创建pod资源
[root@master01 ~]# cat myapp.yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: myapp namespace: default labels: app: myapp spec: replicas: 2 selector: matchLabels: app: myapp template: metadata: name: myapp-pod labels: app: myapp spec: containers: - name: myapp image: ikubernetes/myapp:v1 resources: requests: cpu: 50m memory: 64Mi limits: cpu: 50m memory: 64Mi --- apiVersion: v1 kind: Service metadata: name: myapp-svc labels: app: myapp namespace: default spec: selector: app: myapp ports: - name: http port: 80 targetPort: 80 type: NodePort [root@master01 ~]#
应用资源清单
[root@master01 ~]# kubectl apply -f myapp.yaml deployment.apps/myapp created service/myapp-svc created [root@master01 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE myapp-779867bcfc-657qr 1/1 Running 0 6s myapp-779867bcfc-txvj8 1/1 Running 0 6s [root@master01 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d myapp-svc NodePort 10.111.14.219 <none> 80:31154/TCP 13s [root@master01 ~]#
查看pod的资源占比情况
[root@master01 ~]# kubectl top pods NAME CPU(cores) MEMORY(bytes) myapp-779867bcfc-657qr 0m 3Mi myapp-779867bcfc-txvj8 0m 3Mi [root@master01 ~]#
提示：现在没有访问对应pod，其cpu指标为0；
使用命令创建hpa资源，监控myapp deploy，指定对应pod的cpu资源使用率达到50%就触发hpa
[root@master01 ~]# kubectl autoscale deploy myapp --min=2 --max=10 --cpu-percent=50 horizontalpodautoscaler.autoscaling/myapp autoscaled [root@master01 ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE myapp Deployment/myapp <unknown>/50% 2 10 0 10s [root@master01 ~]# kubectl describe hpa/myapp Name: myapp Namespace: default Labels: <none> Annotations: <none> CreationTimestamp: Mon, 18 Jan 2021 15:26:49 +0800 Reference: Deployment/myapp Metrics: ( current / target ) resource cpu on pods (as a percentage of request): 0% (0) / 50% Min replicas: 2 Max replicas: 10 Deployment pods: 2 current / 2 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request) ScalingLimited False DesiredWithinRange the desired count is within the acceptable range Events: <none> [root@master01 ~]#
验证：使用ab压测工具，对pod进行压力访问，看看对应pod cpu资源使用率大于50%，对应pod是否会扩展？
安装ab工具
yum install httpd-tools -y
使用外部主机对pod svc 进行压力访问
查看pod的资源占比情况
[root@master01 ~]# kubectl top pods NAME CPU(cores) MEMORY(bytes) myapp-779867bcfc-657qr 51m 4Mi myapp-779867bcfc-txvj8 34m 4Mi [root@master01 ~]#
提示：可以看到对应pod的cpu资源都大于限制的资源上限的50%；这里需要注意hpa扩展pod它有一定的延迟，不是立刻马上就扩展；
查看对应hpa资源的详情
提示：hpa详情告诉我们对应pod扩展到7个；
查看pod数量是否被扩展到7个？
[root@master01 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE myapp-779867bcfc-657qr 1/1 Running 0 16m myapp-779867bcfc-7c4dt 1/1 Running 0 3m27s myapp-779867bcfc-b2jmn 1/1 Running 0 3m27s myapp-779867bcfc-fmw7v 1/1 Running 0 2m25s myapp-779867bcfc-hxhj2 1/1 Running 0 2m25s myapp-779867bcfc-txvj8 1/1 Running 0 16m myapp-779867bcfc-xvh58 1/1 Running 0 2m25s [root@master01 ~]#
提示：可以看到对应pod被控制到7个；
查看对应pod的资源占比是否还高于限制的50%？
[root@master01 ~]# kubectl top pods NAME CPU(cores) MEMORY(bytes) myapp-779867bcfc-657qr 49m 4Mi myapp-779867bcfc-7c4dt 25m 4Mi myapp-779867bcfc-b2jmn 36m 4Mi myapp-779867bcfc-fmw7v 42m 4Mi myapp-779867bcfc-hxhj2 46m 3Mi myapp-779867bcfc-txvj8 21m 4Mi myapp-779867bcfc-xvh58 49m 4Mi [root@master01 ~]#
提示：可以看到对应pod的cpu使用率还是高于限制的50%，说明扩展到pod数量不能够响应对应的请求，此时hpa还会扩展；
查看hpa详情，看看是否又一次扩展pod的数量？
提示：可以看到对应pod被扩展到10个，但是对应cpu资源使用率为94%，此时pod数量已经到达上限，即便对应指标数据还是大于指定的阀值，它也不会扩展；
查看pod数量是否为10个？
[root@master01 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE myapp-779867bcfc-57zw7 1/1 Running 0 5m39s myapp-779867bcfc-657qr 1/1 Running 0 23m myapp-779867bcfc-7c4dt 1/1 Running 0 10m myapp-779867bcfc-b2jmn 1/1 Running 0 10m myapp-779867bcfc-dvq6k 1/1 Running 0 5m39s myapp-779867bcfc-fmw7v 1/1 Running 0 9m45s myapp-779867bcfc-hxhj2 1/1 Running 0 9m45s myapp-779867bcfc-n8lmf 1/1 Running 0 5m39s myapp-779867bcfc-txvj8 1/1 Running 0 23m myapp-779867bcfc-xvh58 1/1 Running 0 9m45s [root@master01 ~]#
停止压测，看看对应pod是否会缩减到最低pod数量为2个呢？
[root@master01 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE myapp-779867bcfc-57zw7 1/1 Running 0 8m8s myapp-779867bcfc-657qr 1/1 Running 0 26m myapp-779867bcfc-7c4dt 1/1 Running 0 13m myapp-779867bcfc-b2jmn 1/1 Running 0 13m myapp-779867bcfc-dvq6k 1/1 Running 0 8m8s myapp-779867bcfc-fmw7v 1/1 Running 0 12m myapp-779867bcfc-hxhj2 1/1 Running 0 12m myapp-779867bcfc-n8lmf 1/1 Running 0 8m8s myapp-779867bcfc-txvj8 1/1 Running 0 26m myapp-779867bcfc-xvh58 1/1 Running 0 12m [root@master01 ~]# kubectl top pods NAME CPU(cores) MEMORY(bytes) myapp-779867bcfc-57zw7 0m 4Mi myapp-779867bcfc-657qr 0m 4Mi myapp-779867bcfc-7c4dt 7m 4Mi myapp-779867bcfc-b2jmn 0m 4Mi myapp-779867bcfc-dvq6k 0m 4Mi myapp-779867bcfc-fmw7v 0m 4Mi myapp-779867bcfc-hxhj2 3m 3Mi myapp-779867bcfc-n8lmf 10m 4Mi myapp-779867bcfc-txvj8 0m 4Mi myapp-779867bcfc-xvh58 0m 4Mi [root@master01 ~]#
提示：pod缩减也是不会立刻执行；从上面信息可以看到停止压测对应pod的cpu资源使用率都降下来了；
再次查看对应pod数量
[root@master01 ~]# kubectl top pods NAME CPU(cores) MEMORY(bytes) myapp-779867bcfc-57zw7 0m 4Mi myapp-779867bcfc-657qr 0m 4Mi [root@master01 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE myapp-779867bcfc-57zw7 1/1 Running 0 13m myapp-779867bcfc-657qr 1/1 Running 0 31m [root@master01 ~]#
提示：可以看到对应pod缩减到最低pod副本数量；
查看hpa的详情
提示：可以看到对应pod的cpu使用率小于50%，它会隔一段时间就缩减对应pod；
示例：使用资源清单定义hpa资源
[root@master01 ~]# cat hpa-demo.yaml apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: hpa-demo spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: myapp minReplicas: 2 maxReplicas: 10 targetCPUUtilizationPercentage: 50 [root@master01 ~]#
提示：以上是hpa v1的资源清单定义示例，其中targetCPUUtilizationPercentage用于指定cpu资源使用率阀值，50表示50%，即达到pod上限的50%对应hpa就会被触发；
应用清单
[root@master01 ~]# kubectl apply -f hpa-demo.yaml horizontalpodautoscaler.autoscaling/hpa-demo created [root@master01 ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE hpa-demo Deployment/myapp <unknown>/50% 2 10 0 8s myapp Deployment/myapp 0%/50% 2 10 2 35m [root@master01 ~]# kubectl describe hpa/hpa-demo Name: hpa-demo Namespace: default Labels: <none> Annotations: <none> CreationTimestamp: Mon, 18 Jan 2021 16:02:25 +0800 Reference: Deployment/myapp Metrics: ( current / target ) resource cpu on pods (as a percentage of request): 0% (0) / 50% Min replicas: 2 Max replicas: 10 Deployment pods: 2 current / 2 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request) ScalingLimited False DesiredWithinRange the desired count is within the acceptable range Events: <none> [root@master01 ~]#
提示：可以看到使用命令创建hpa和使用资源清单创建hpa其创建出来的hpa都是一样的；以上是hpa v1的使用示例和相关说明；使用命令创建hpa，只能创建v1的hpa；v2必须使用资源清单，明确指定对应hpa的群组版本；
使用自定义资源指标定义hpa
部署自定义资源指标服务器
下载部署清单
[root@master01 ~]# mkdir custom-metrics-server [root@master01 ~]# cd custom-metrics-server [root@master01 custom-metrics-server]# git clone https://github.com/stefanprodan/k8s-prom-hpa Cloning into 'k8s-prom-hpa'... remote: Enumerating objects: 223, done. remote: Total 223 (delta 0), reused 0 (delta 0), pack-reused 223 Receiving objects: 100% (223/223), 102.23 KiB | 14.00 KiB/s, done. Resolving deltas: 100% (117/117), done. [root@master01 custom-metrics-server]# ls k8s-prom-hpa [root@master01 custom-metrics-server]#
查看custom-metrics-server的部署清单
[root@master01 custom-metrics-server]# cd k8s-prom-hpa/ [root@master01 k8s-prom-hpa]# ls custom-metrics-api diagrams ingress LICENSE Makefile metrics-server namespaces.yaml podinfo prometheus README.md [root@master01 k8s-prom-hpa]# cd custom-metrics-api/ [root@master01 custom-metrics-api]# ls custom-metrics-apiserver-auth-delegator-cluster-role-binding.yaml custom-metrics-apiservice.yaml custom-metrics-apiserver-auth-reader-role-binding.yaml custom-metrics-cluster-role.yaml custom-metrics-apiserver-deployment.yaml custom-metrics-config-map.yaml custom-metrics-apiserver-resource-reader-cluster-role-binding.yaml custom-metrics-resource-reader-cluster-role.yaml custom-metrics-apiserver-service-account.yaml hpa-custom-metrics-cluster-role-binding.yaml custom-metrics-apiserver-service.yaml [root@master01 custom-metrics-api]# cat custom-metrics-apiserver-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: labels: app: custom-metrics-apiserver name: custom-metrics-apiserver namespace: monitoring spec: replicas: 1 selector: matchLabels: app: custom-metrics-apiserver template: metadata: labels: app: custom-metrics-apiserver name: custom-metrics-apiserver spec: serviceAccountName: custom-metrics-apiserver containers: - name: custom-metrics-apiserver image: quay.io/coreos/k8s-prometheus-adapter-amd64:v0.4.1 args: - /adapter - --secure-port=6443 - --tls-cert-file=/var/run/serving-cert/serving.crt - --tls-private-key-file=/var/run/serving-cert/serving.key - --logtostderr=true - --prometheus-url=http://prometheus.monitoring.svc:9090/ - --metrics-relist-interval=30s - --v=10 - --config=/etc/adapter/config.yaml ports: - containerPort: 6443 volumeMounts: - mountPath: /var/run/serving-cert name: volume-serving-cert readOnly: true - mountPath: /etc/adapter/ name: config readOnly: true volumes: - name: volume-serving-cert secret: secretName: cm-adapter-serving-certs - name: config configMap: name: adapter-config [root@master01 custom-metrics-api]#
提示：上述清单中明确定义了把自定义指标服务器部署到monitoring名称空间下，对应server的启动还挂在了secret证书；所以应用上述清单前，我们要先创建名称空间和secret；在创建secret前还要先准备好证书和私钥；这里还需要注意custom-metrics-server是连接Prometheus server，把对应自定义数据通过apiservice注册到对应原生apiserver上，供k8s组件使用，所以这里要注意对应Prometheus的地址；
创建monitoring名称空间
[root@master01 custom-metrics-api]# cd .. [root@master01 k8s-prom-hpa]# ls custom-metrics-api diagrams ingress LICENSE Makefile metrics-server namespaces.yaml podinfo prometheus README.md [root@master01 k8s-prom-hpa]# cat namespaces.yaml --- apiVersion: v1 kind: Namespace metadata: name: monitoring [root@master01 k8s-prom-hpa]# kubectl apply -f namespaces.yaml namespace/monitoring created [root@master01 k8s-prom-hpa]# kubectl get ns NAME STATUS AGE default Active 41d ingress-nginx Active 27d kube-node-lease Active 41d kube-public Active 41d kube-system Active 41d kubernetes-dashboard Active 16d mongodb Active 4d20h monitoring Active 4s [root@master01 k8s-prom-hpa]#
生成serving.key和serving.csr
[root@master01 k8s-prom-hpa]# cd /etc/kubernetes/pki/ [root@master01 pki]# ls apiserver.crt apiserver.key ca.crt etcd front-proxy-client.crt sa.pub tom.key apiserver-etcd-client.crt apiserver-kubelet-client.crt ca.key front-proxy-ca.crt front-proxy-client.key tom.crt apiserver-etcd-client.key apiserver-kubelet-client.key ca.srl front-proxy-ca.key sa.key tom.csr [root@master01 pki]# openssl genrsa -out serving.key 2048 Generating RSA private key, 2048 bit long modulus .............................................................................................................................................................+++ ..............................+++ e is 65537 (0x10001) [root@master01 pki]# openssl req -new -key ./serving.key -out ./serving.csr -subj "/CN=serving" [root@master01 pki]# ll total 80 -rw-r--r-- 1 root root 1277 Dec 8 14:38 apiserver.crt -rw-r--r-- 1 root root 1135 Dec 8 14:38 apiserver-etcd-client.crt -rw------- 1 root root 1679 Dec 8 14:38 apiserver-etcd-client.key -rw------- 1 root root 1679 Dec 8 14:38 apiserver.key -rw-r--r-- 1 root root 1143 Dec 8 14:38 apiserver-kubelet-client.crt -rw------- 1 root root 1679 Dec 8 14:38 apiserver-kubelet-client.key -rw-r--r-- 1 root root 1066 Dec 8 14:38 ca.crt -rw------- 1 root root 1675 Dec 8 14:38 ca.key -rw-r--r-- 1 root root 17 Jan 17 13:03 ca.srl drwxr-xr-x 2 root root 162 Dec 8 14:38 etcd -rw-r--r-- 1 root root 1078 Dec 8 14:38 front-proxy-ca.crt -rw------- 1 root root 1675 Dec 8 14:38 front-proxy-ca.key -rw-r--r-- 1 root root 1103 Dec 8 14:38 front-proxy-client.crt -rw------- 1 root root 1679 Dec 8 14:38 front-proxy-client.key -rw------- 1 root root 1679 Dec 8 14:38 sa.key -rw------- 1 root root 451 Dec 8 14:38 sa.pub -rw-r--r-- 1 root root 887 Jan 18 16:54 serving.csr -rw-r--r-- 1 root root 1679 Jan 18 16:54 serving.key -rw-r--r-- 1 root root 993 Dec 30 00:29 tom.crt -rw-r--r-- 1 root root 907 Dec 30 00:27 tom.csr -rw-r--r-- 1 root root 1675 Dec 30 00:21 tom.key [root@master01 pki]#
用kubenetes CA的key和证书为custom-metrics-server签署证书
[root@master01 pki]# openssl x509 -req -in serving.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out serving.crt -days 3650 Signature ok subject=/CN=serving Getting CA Private Key [root@master01 pki]# ll serving.crt -rw-r--r-- 1 root root 977 Jan 18 16:55 serving.crt [root@master01 pki]#
在monitoring名称空间下创建secret资源
[root@master01 pki]# kubectl create secret generic cm-adapter-serving-certs --from-file=./serving.key --from-file=./serving.crt -n monitoring secret/cm-adapter-serving-certs created [root@master01 pki]# kubectl get secret -n monitoring NAME TYPE DATA AGE cm-adapter-serving-certs Opaque 2 10s default-token-k64tz kubernetes.io/service-account-token 3 2m27s [root@master01 pki]# kubectl describe secret/cm-adapter-serving-certs -n monitoring Name: cm-adapter-serving-certs Namespace: monitoring Labels: <none> Annotations: <none> Type: Opaque Data ==== serving.crt: 977 bytes serving.key: 1679 bytes [root@master01 pki]#
提示：这里一定要使用generic类型创建secret，保持对应的名称为serving.key和serving.crt；创建secret的名称，必须同custom-metrics部署清单中的名称保持一致；
部署prometheus
[root@master01 pki]# cd /root/custom-metrics-server/k8s-prom-hpa/prometheus/ [root@master01 prometheus]# ls prometheus-cfg.yaml prometheus-dep.yaml prometheus-rbac.yaml prometheus-svc.yaml [root@master01 prometheus]# cat prometheus-dep.yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: prometheus namespace: monitoring spec: replicas: 1 selector: matchLabels: app: prometheus template: metadata: labels: app: prometheus annotations: prometheus.io/scrape: 'false' spec: serviceAccountName: prometheus containers: - name: prometheus image: prom/prometheus:v2.1.0 imagePullPolicy: Always command: - prometheus - --config.file=/etc/prometheus/prometheus.yml - --storage.tsdb.retention=1h ports: - containerPort: 9090 protocol: TCP resources: limits: memory: 2Gi volumeMounts: - mountPath: /etc/prometheus/prometheus.yml name: prometheus-config subPath: prometheus.yml volumes: - name: prometheus-config configMap: name: prometheus-config items: - key: prometheus.yml path: prometheus.yml mode: 0644 [root@master01 prometheus]#
更改rbac资源清单中的群组版本为 rbac.authorization.k8s.io/v1
[root@master01 prometheus]# cat prometheus-rbac.yaml --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prometheus rules: - apiGroups: [""] resources: - nodes - nodes/proxy - services - endpoints - pods verbs: ["get", "list", "watch"] - apiGroups: - extensions resources: - ingresses verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics"] verbs: ["get"] --- apiVersion: v1 kind: ServiceAccount metadata: name: prometheus namespace: monitoring --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: prometheus roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prometheus subjects: - kind: ServiceAccount name: prometheus namespace: monitoring [root@master01 prometheus]#
应用prometheus目录下的所有资源清单
[root@master01 prometheus]# kubectl apply -f . configmap/prometheus-config created deployment.apps/prometheus created clusterrole.rbac.authorization.k8s.io/prometheus created serviceaccount/prometheus created clusterrolebinding.rbac.authorization.k8s.io/prometheus created service/prometheus created [root@master01 prometheus]# kubectl get pods -n monitoring NAME READY STATUS RESTARTS AGE prometheus-5c5dc6d6d4-drrht 1/1 Running 0 26s [root@master01 prometheus]# kubectl get svc -n monitoring NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE prometheus NodePort 10.99.1.110 <none> 9090:31190/TCP 35s [root@master01 prometheus]#
部署自定义指标服务，应用custom-metrics-server目录下的所有资源清单
[root@master01 prometheus]# cd ../custom-metrics-api/ [root@master01 custom-metrics-api]# ls custom-metrics-apiserver-auth-delegator-cluster-role-binding.yaml custom-metrics-apiservice.yaml custom-metrics-apiserver-auth-reader-role-binding.yaml custom-metrics-cluster-role.yaml custom-metrics-apiserver-deployment.yaml custom-metrics-config-map.yaml custom-metrics-apiserver-resource-reader-cluster-role-binding.yaml custom-metrics-resource-reader-cluster-role.yaml custom-metrics-apiserver-service-account.yaml hpa-custom-metrics-cluster-role-binding.yaml custom-metrics-apiserver-service.yaml [root@master01 custom-metrics-api]# kubectl apply -f . clusterrolebinding.rbac.authorization.k8s.io/custom-metrics:system:auth-delegator created rolebinding.rbac.authorization.k8s.io/custom-metrics-auth-reader created deployment.apps/custom-metrics-apiserver created clusterrolebinding.rbac.authorization.k8s.io/custom-metrics-resource-reader created serviceaccount/custom-metrics-apiserver created service/custom-metrics-apiserver created apiservice.apiregistration.k8s.io/v1beta1.custom.metrics.k8s.io created clusterrole.rbac.authorization.k8s.io/custom-metrics-server-resources created configmap/adapter-config created clusterrole.rbac.authorization.k8s.io/custom-metrics-resource-reader created clusterrolebinding.rbac.authorization.k8s.io/hpa-controller-custom-metrics created [root@master01 custom-metrics-api]# kubectl get pods -n monitoring NAME READY STATUS RESTARTS AGE custom-metrics-apiserver-754dfc87c7-cdhqj 1/1 Running 0 18s prometheus-5c5dc6d6d4-drrht 1/1 Running 0 6m9s [root@master01 custom-metrics-api]# kubectl get svc -n monitoring NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE custom-metrics-apiserver ClusterIP 10.99.245.190 <none> 443/TCP 31s prometheus NodePort 10.99.1.110 <none> 9090:31190/TCP 6m21s [root@master01 custom-metrics-api]#
提示：应用上述清单前，请把所有rbac.authorization.k8s.io/v1beta1更改为rbac.authorization.k8s.io/v1，把apiservice中的版本为apiregistration.k8s.io/v1；如果是1.17之前的k8s，不用修改；
验证：查看原生apiserver是否有custom.metrics.k8s.io的群组注册进来？
[root@master01 custom-metrics-api]# kubectl api-versions|grep custom custom.metrics.k8s.io/v1beta1 [root@master01 custom-metrics-api]#
验证：访问对应群组，看看是否能够请求到自定义资源指标？
[root@master01 custom-metrics-api]# kubectl get --raw "/apis/metrics.k8s.io/v1beta1/" | jq . { "kind": "APIResourceList", "apiVersion": "v1", "groupVersion": "metrics.k8s.io/v1beta1", "resources": [ { "name": "nodes", "singularName": "", "namespaced": false, "kind": "NodeMetrics", "verbs": [ "get", "list" ] }, { "name": "pods", "singularName": "", "namespaced": true, "kind": "PodMetrics", "verbs": [ "get", "list" ] } ] } [root@master01 custom-metrics-api]#
提示：如果访问对应群组能够响应数据，表示自定义资源指标服务器没有问题；
示例：创建podinfo pod，该pod输出http_requests资源指标
[root@master01 custom-metrics-api]# cd .. [root@master01 k8s-prom-hpa]# ls custom-metrics-api diagrams ingress LICENSE Makefile metrics-server namespaces.yaml podinfo prometheus README.md [root@master01 k8s-prom-hpa]# cd podinfo/ [root@master01 podinfo]# ls podinfo-dep.yaml podinfo-hpa-custom.yaml podinfo-hpa.yaml podinfo-ingress.yaml podinfo-svc.yaml [root@master01 podinfo]# cat podinfo-dep.yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: podinfo spec: selector: matchLabels: app: podinfo replicas: 2 template: metadata: labels: app: podinfo annotations: prometheus.io/scrape: "true" spec: containers: - name: podinfod image: stefanprodan/podinfo:0.0.1 imagePullPolicy: Always command: - ./podinfo - -port=9898 - -logtostderr=true - -v=2 volumeMounts: - name: metadata mountPath: /etc/podinfod/metadata readOnly: true ports: - containerPort: 9898 protocol: TCP readinessProbe: httpGet: path: /readyz port: 9898 initialDelaySeconds: 1 periodSeconds: 2 failureThreshold: 1 livenessProbe: httpGet: path: /healthz port: 9898 initialDelaySeconds: 1 periodSeconds: 3 failureThreshold: 2 resources: requests: memory: "32Mi" cpu: "1m" limits: memory: "256Mi" cpu: "100m" volumes: - name: metadata downwardAPI: items: - path: "labels" fieldRef: fieldPath: metadata.labels - path: "annotations" fieldRef: fieldPath: metadata.annotations [root@master01 podinfo]# cat podinfo-svc.yaml --- apiVersion: v1 kind: Service metadata: name: podinfo labels: app: podinfo spec: type: NodePort ports: - port: 9898 targetPort: 9898 nodePort: 31198 protocol: TCP selector: app: podinfo [root@master01 podinfo]#
应用资源清单
[root@master01 podinfo]# kubectl apply -f podinfo-dep.yaml,./podinfo-svc.yaml deployment.apps/podinfo created service/podinfo created [root@master01 podinfo]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d myapp-svc NodePort 10.111.14.219 <none> 80:31154/TCP 4h35m podinfo NodePort 10.111.10.211 <none> 9898:31198/TCP 17s [root@master01 podinfo]# kubectl get pods NAME READY STATUS RESTARTS AGE myapp-779867bcfc-57zw7 1/1 Running 0 4h18m myapp-779867bcfc-657qr 1/1 Running 0 4h36m podinfo-56874dc7f8-5rb9q 1/1 Running 0 40s podinfo-56874dc7f8-t6jgn 1/1 Running 0 40s [root@master01 podinfo]#
验证：访问podinfo svc，看看对应pod是否能够正常访问？
[root@master01 podinfo]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d myapp-svc NodePort 10.111.14.219 <none> 80:31154/TCP 4h37m podinfo NodePort 10.111.10.211 <none> 9898:31198/TCP 116s [root@master01 podinfo]# curl 10.111.10.211:9898 runtime: arch: amd64 external_ip: "" max_procs: "4" num_cpu: "4" num_goroutine: "9" os: linux version: go1.9.2 labels: app: podinfo pod-template-hash: 56874dc7f8 annotations: cni.projectcalico.org/podIP: 10.244.3.133/32 cni.projectcalico.org/podIPs: 10.244.3.133/32 kubernetes.io/config.seen: 2021-01-18T19:57:31.325293640+08:00 kubernetes.io/config.source: api prometheus.io/scrape: "true" environment: HOME: /root HOSTNAME: podinfo-56874dc7f8-5rb9q KUBERNETES_PORT: tcp://10.96.0.1:443 KUBERNETES_PORT_443_TCP: tcp://10.96.0.1:443 KUBERNETES_PORT_443_TCP_ADDR: 10.96.0.1 KUBERNETES_PORT_443_TCP_PORT: "443" KUBERNETES_PORT_443_TCP_PROTO: tcp KUBERNETES_SERVICE_HOST: 10.96.0.1 KUBERNETES_SERVICE_PORT: "443" KUBERNETES_SERVICE_PORT_HTTPS: "443" MYAPP_SVC_PORT: tcp://10.111.14.219:80 MYAPP_SVC_PORT_80_TCP: tcp://10.111.14.219:80 MYAPP_SVC_PORT_80_TCP_ADDR: 10.111.14.219 MYAPP_SVC_PORT_80_TCP_PORT: "80" MYAPP_SVC_PORT_80_TCP_PROTO: tcp MYAPP_SVC_SERVICE_HOST: 10.111.14.219 MYAPP_SVC_SERVICE_PORT: "80" MYAPP_SVC_SERVICE_PORT_HTTP: "80" PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PODINFO_PORT: tcp://10.111.10.211:9898 PODINFO_PORT_9898_TCP: tcp://10.111.10.211:9898 PODINFO_PORT_9898_TCP_ADDR: 10.111.10.211 PODINFO_PORT_9898_TCP_PORT: "9898" PODINFO_PORT_9898_TCP_PROTO: tcp PODINFO_SERVICE_HOST: 10.111.10.211 PODINFO_SERVICE_PORT: "9898" [root@master01 podinfo]#
验证：访问apiserver，看看对应pod输出的http_requests资源指标是否能够被访问到？
[root@master01 podinfo]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests" | jq . { "kind": "MetricValueList", "apiVersion": "custom.metrics.k8s.io/v1beta1", "metadata": { "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests" }, "items": [ { "describedObject": { "kind": "Pod", "namespace": "default", "name": "podinfo-56874dc7f8-5rb9q", "apiVersion": "/v1" }, "metricName": "http_requests", "timestamp": "2021-01-18T12:01:41Z", "value": "911m" }, { "describedObject": { "kind": "Pod", "namespace": "default", "name": "podinfo-56874dc7f8-t6jgn", "apiVersion": "/v1" }, "metricName": "http_requests", "timestamp": "2021-01-18T12:01:41Z", "value": "888m" } ] } [root@master01 podinfo]#
提示：可以看到现在用kubectl 工具可以在apiserver上访问到对应pod提供的自定义指标数据；
示例：根据自定义指标数据，定义hpa资源
[root@master01 podinfo]# cat podinfo-hpa-custom.yaml --- apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: podinfo spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: podinfo minReplicas: 2 maxReplicas: 10 metrics: - type: Pods pods: metric: name: http_requests target: type: AverageValue averageValue: 10 [root@master01 podinfo]#
提示：使用自定义资源指标，对应hpa的群组必须为autoscale/v2beta2；对应自定义指标用metrics字段给定；type用来描述对应自定义指标数据是什么类型，pod表示是pod自身提供的自定义指标数据；上述资源清单表示引用pod自身的自定义指标数据，其名称为http_requests；对该指标数据的平均值做监控，如果对应指标平均值大于10，则触发hpa对其扩展，当对应指标数据小于10，对应hpa会对应进行缩减操作；
应用资源清单
[root@master01 podinfo]# kubectl apply -f podinfo-hpa-custom.yaml horizontalpodautoscaler.autoscaling/podinfo created [root@master01 podinfo]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE hpa-demo Deployment/myapp 0%/50% 2 10 2 4h1m myapp Deployment/myapp 0%/50% 2 10 2 4h37m podinfo Deployment/podinfo <unknown>/10 2 10 0 6s [root@master01 podinfo]# kubectl describe hpa/podinfo Name: podinfo Namespace: default Labels: <none> Annotations: <none> CreationTimestamp: Mon, 18 Jan 2021 20:04:14 +0800 Reference: Deployment/podinfo Metrics: ( current / target ) "http_requests" on pods: 899m / 10 Min replicas: 2 Max replicas: 10 Deployment pods: 2 current / 2 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from pods metric http_requests ScalingLimited False DesiredWithinRange the desired count is within the acceptable range Events: <none> [root@master01 podinfo]#
对podinfo 进行压测，看看对应hpa是否能够自动扩展？
提示：可以看到对应pod能够被对应的hpa通过自定义指标来扩展pod数量；
停止压测，看看对应pod是否会自动缩减至最低数量？ 
提示：可以看到停止压测以后，对应的指标数据降低下来，对应的pod也随之缩减到最低副本数量；以上就是hpa v2的简单使用方式，更多示例和说明请参考官方文档https://kubernetes.io/zh/docs/tasks/run-application/horizontal-pod-autoscale/；
