Kubernetes heapster

发布时间：2023-09-06 02:14责任编辑：傅花花关键词：暂无标签

Heapster是容器集群监控和性能分析工具，HPA、Dashborad、Kubectl top都依赖于heapster收集的数据。

但是Heapster从kubernetes 1.8以后已经被遗弃了...... 被metrics-server所替代......

kubernetes 1.11 中部署Heapster 1.5.4版本的过程

Heapster 部署yaml文件

apiVersion: v1kind: ServiceAccountmetadata: ?name: heapster ?namespace: kube-system---kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata: ?name: heapsterroleRef: ?apiGroup: rbac.authorization.k8s.io ?kind: ClusterRole ?name: system:heapstersubjects:- kind: ServiceAccount ?name: heapster ?namespace: kube-system---apiVersion: apps/v1kind: Deploymentmetadata: ?name: heapster ?namespace: kube-systemspec: ?replicas: 1 ?template: ???metadata: ?????labels: ???????task: monitoring ???????k8s-app: heapster ???spec: ?????serviceAccountName: heapster ?????containers: ?????- name: heapster ???????# image: k8s.gcr.io/heapster-amd64:v1.5.4 将默认google的官方镜像替换为阿里云镜像，否则你懂得 ???????image: registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-amd64:v1.5.4 ???????command: ???????- /heapster ???????- --source=kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true---apiVersion: v1kind: Servicemetadata: ?labels: ???task: monitoring ???# For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons) ???# If you are NOT using this as an addon, you should comment out this line. ???kubernetes.io/cluster-service: ‘true‘ ???kubernetes.io/name: Heapster ?name: heapster ?namespace: kube-systemspec: ?ports: ?- port: 80 ???targetPort: 8082 ?selector: ???k8s-app: heapster

heapster.yaml

heapster 启动参数说明：

inClusterConfig - Use kube config in service accounts associated with Heapster‘s namespace. (default: true)
kubeletPort - kubelet port to use (default: 10255)
kubeletHttps - whether to use https to connect to kubelets (default: false)
insecure - whether to trust Kubernetes certificates (default: false)
auth - client auth file to use. Set auth if the service accounts are not usable.
useServiceAccount - whether to use the service account token if one is mounted at /var/run/secrets/kubernetes.io/serviceaccount/token (default: false)

使用： kubectl apply -f heapster.yaml 部署

部署后查看log日志，发现一直提示“403 Forbidden”, response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"

[root@node01 heapster-yaml]# kubectl logs --namespace=kube-system heapster-868d5cd7f-855k7 I0912 14:36:48.175115 ??????1 heapster.go:78] /heapster --source=kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=trueI0912 14:36:48.175174 ??????1 heapster.go:79] Heapster version v1.5.4I0912 14:36:48.175366 ??????1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1I0912 14:36:48.175393 ??????1 configs.go:62] Using kubelet port 10250I0912 14:36:48.185828 ??????1 heapster.go:202] Starting with Metric SinkI0912 14:36:48.199515 ??????1 heapster.go:112] Starting heapster on port 8082E0912 14:37:05.000327 ??????1 kubelet.go:288] node node01 is not readyE0912 14:38:05.016044 ??????1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"W0912 14:38:25.000643 ??????1 manager.go:152] Failed to get all responses in time (got 0/1)E0912 14:39:05.008534 ??????1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"W0912 14:39:25.000674 ??????1 manager.go:152] Failed to get all responses in time (got 0/1)E0912 14:40:05.009955 ??????1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"W0912 14:40:25.001231 ??????1 manager.go:152] Failed to get all responses in time (got 0/1)E0912 14:41:05.017198 ??????1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"W0912 14:41:25.000846 ??????1 manager.go:152] Failed to get all responses in time (got 0/1)

查看ClusterRole: system:heapster的权限，发现的确没有针对Resource: nodes/stats 的create权限

[root@node01 heapster-yaml]# kubectl describe clusterrole system:heapsterName: ????????system:heapsterLabels: ??????kubernetes.io/bootstrapping=rbac-defaultsAnnotations: ?kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{"rbac.authorization.kubernetes.io/autoupdate":"true"},"lab... ?????????????rbac.authorization.kubernetes.io/autoupdate=truePolicyRule: ?Resources ??????????????Non-Resource URLs ?Resource Names ?Verbs ?--------- ??????????????----------------- ?-------------- ?----- ?events ?????????????????[] ????????????????[] ?????????????[get list watch] ?namespaces ?????????????[] ????????????????[] ?????????????[get list watch] ?nodes ??????????????????[] ????????????????[] ?????????????[get list watch] ?pods ???????????????????[] ????????????????[] ?????????????[get list watch] ?deployments.extensions ?[] ????????????????[] ?????????????[get list watch]

修改ClusterRole: system:heapster的权限:

1. 查看system:heapster yaml格式, 保存为 heapster-clusterrole.yaml

yaml

[root@node01 heapster-yaml]# kubectl get clusterrole system:heapster -o yamlapiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: ?annotations: ???kubectl.kubernetes.io/last-applied-configuration: | ?????{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{"rbac.authorization.kubernetes.io/autoupdate":"true"},"labels":{"kubernetes.io/bootstrapping":"rbac-defaults"},"name":"system:heapster","namespace":""},"rules":[{"apiGroups":[""],"resources":["events","namespaces","nodes","pods","nodes/stats"],"verbs":["create","get","list","watch"]},{"apiGroups":["extensions"],"resources":["deployments"],"verbs":["get","list","watch"]}]} ???rbac.authorization.kubernetes.io/autoupdate: "true" ?creationTimestamp: 2018-08-26T02:26:14Z ?labels: ???kubernetes.io/bootstrapping: rbac-defaults ?name: system:heapster ?resourceVersion: "139000" ?selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Aheapster ?uid: 67ef3689-a8d7-11e8-a891-000c29b52823rules:- apiGroups: ?- "" ?resources: ?- events ?- namespaces ?- nodes ?- pods ?verbs: ?- get ?- list ?- watch- apiGroups: ?- extensions ?resources: ?- deployments ?verbs: ?- get ?- list ?- watch

2. 添加Resource: nodes/stats的create权限，并执行 kubectl apply -f heapster-clusterrole.yaml

yaml

apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: ?annotations: ???rbac.authorization.kubernetes.io/autoupdate: "true" ?labels: ???kubernetes.io/bootstrapping: rbac-defaults ?name: system:heapsterrules:- apiGroups: ?- "" ?resources: ?- events ?- namespaces ?- nodes ?- pods ?- nodes/stats ?verbs: ?- create ?- get ?- list ?- watch- apiGroups: ?- extensions ?resources: ?- deployments ?verbs: ?- get ?- list ?- watch

3. 删除heapster重新部署

　　kubectl delete -f heapster.yaml

　　kubectl apply -f heapster.yaml

4. 重新部署后，查看log没有任何报错了

[root@node01 heapster-yaml]# kubectl logs --namespace=kube-system heapster-868d5cd7f-8zgxq I0912 15:02:38.926068 ??????1 heapster.go:78] /heapster --source=kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=trueI0912 15:02:38.926128 ??????1 heapster.go:79] Heapster version v1.5.4I0912 15:02:38.926565 ??????1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1I0912 15:02:38.926647 ??????1 configs.go:62] Using kubelet port 10250I0912 15:02:38.938333 ??????1 heapster.go:202] Starting with Metric SinkI0912 15:02:38.949215 ??????1 heapster.go:112] Starting heapster on port 8082

5. 功能测试

部署前：

[root@node01 heapster-yaml]# kubectl top podError from server (NotFound): the server could not find the requested resource (get services http:heapster:)

部署后：

[root@node01 tomcat-mysql-yaml]# kubectl top podNAME ?????????????????????CPU(cores) ??MEMORY(bytes) ??lxcfs-j8bpd ??????????????0m ??????????4Mi ????????????mysql-85c85c5668-fr6j7 ???0m ??????????458Mi ??????????tomcat-78c9778858-424gc ??1m ??????????93Mi ???????????tomcat-78c9778858-42xz4 ??1m ??????????97Mi ???????????tomcat-78c9778858-6t7cd ??1m ??????????93Mi ???????????

Kubernetes heapster

kubernetes 1.11 中部署Heapster 1.5.4版本的过程

知识推荐