prometheus它是一个主动拉取的数据库，在K8S中应该展示图形的grafana数据实例化要保存下来，使用分布式文件系统加动态PV，但是在本测试环境中使用本地磁盘，安装采集数据的agent使用DaemonSet来部署，DaemonSet的特性就是在每个node上部署一个服务进程，这一切都是自动的部署。
此处只讲如何用prometheus来监控K8S集群，关于prometheus的知识参考官方文档。
部署前提： 准备好所需要的文件
$ls-lPrometheus/prometheus#:/data/Prometheus/prometheus#ls-ltotal28drwxr-xr-x2rootroot4096Jan1502:53grafanadrwxr-xr-x2rootroot4096Jan1503:11kube-state-metrics-rw-r--r--1rootroot60Jan1406:48namespace.yamldrwxr-xr-x2rootroot4096Jan1503:22node-directory-size-metricsdrwxr-xr-x2rootroot4096Jan1503:02node-exporterdrwxr-xr-x2rootroot4096Jan1502:55prometheusdrwxr-xr-x2rootroot4096Jan1502:37rbac$lsgrafana/grafana-configmap.yamlgrafana-core-deployment.yamlgrafana-import-dashboards-job.yamlgrafana-pvc-claim.yamlgrafana-pvc-volume.yamlgrafana-service.yaml$lsprometheus/configmap.yamldeployment.yamlprometheus-rules.yamlservice.yaml
grafana和prometheus 都是部署文件，node-exporter、kube-state-metrics、node-directory-size-metrics这三个是采集器，相当于prometheus的agent
文件准备好了，现在开始一步一步来部署：
1，,创建所需Namespace
因为prometheus 部署的所有的deploy、pod、svc都是在monitoring完成的，所以需要事先创建之。
$catnamespace.yamlapiVersion:v1kind:Namespacemetadata:name:monitoring$kubectlcreate-fnamespace.yamlnamespace"monitoring"created
2，创建grafana的pv、 pvc
grafana#catgrafana-pvc-volume.yamlkind:PersistentVolumeapiVersion:v1metadata:name:grafana-pv-volumelabels:type:localspec:storageClassName:grafana-pv-volumecapacity:storage:10GiaccessModes:-ReadWriteOncepersistentVolumeReclaimPolicy:RecyclehostPath:path:"/data/volume/grafana"grafana#catgrafana-pvc-claim.yamlkind:PersistentVolumeClaimapiVersion:v1metadata:name:grafana-pvc-volumenamespace:"monitoring"spec:accessModes:-ReadWriteOnceresources:requests:storage:5GistorageClassName:grafana-pv-volume$kubectlcreate-fgrafana/grafana-pvc-volume.yaml-fgrafana/grafana-pvc-claim.yamlpersistentvolume"grafana-pv-volume"createdpersistentvolumeclaim"grafana-pvc-volume"created$kubectlgetpvc-nmonitoringNAMESTATUSVOLUMECAPACITYACCESSMODESSTORAGECLASSAGEgrafana-pvc-volumeBoundgrafana-pv-volume10GiRWOgrafana-pv-volume52s状态bound已绑定到了grafana-pv-volume
3，创建grafana应用，这些应用都是第三方的，都会有自已的配置，通过configmap来定义
grafana#lsgrafana-configmap.yamlgrafana-core-deployment.yamlgrafana-import-dashboards-job.yamlgrafana-pvc-claim.yamlgrafana-pvc-volume.yamlgrafana-service.yamlgrafana#kubectlcreate-f./#grafana目录下所有文件都创建configmap"grafana-import-dashboards"createddeployment"grafana-core"createdjob"grafana-import-dashboards"createdservice"grafana"createdgrafana#kubectlgetdeployment,pod-nmonitoringNAMEDESIREDCURRENTUP-TO-DATEAVAILABLEAGEdeploy/grafana-core11101mNAMEREADYSTATUSRESTARTSAGEpo/grafana-core-9c7f66868-7q8lx0/1ContainerCreating01m运行po/grafana-core容器时会下载镜像：grafana/grafana:4.2.0
grafana创建的应用 简单的自已描述了下：
grafana-pv-volume=/data/volume/grafana=10Ggrafana-pvc-volume=5G--->grafana-pv-volume---configmap=grafana-import-dashboardsJob=grafana-import-dashboardsDeployment=grafana-corereplicas:1containers=grafana-coremount:grafana-pvc-volume:/varservice=grafanaport:3000=nodePort:30161(3000是grafana服务的默认端口)
4， 现在grafana的核心应用已部署好了，现在来部署prometheus的RBAC
prometheus/rbac#lsgrant_serviceAccount.shprometheus_rbac.yaml#先创建RBAC文件：prometheus/rbac#kubectlcreate-fprometheus_rbac.yamlclusterrolebinding"prometheus-k8s"createdclusterrolebinding"kube-state-metrics"createdclusterrole"kube-state-metrics"createdserviceaccount"kube-state-metrics"createdclusterrolebinding"prometheus"createdclusterrole"prometheus"createdserviceaccount"prometheus-k8s"createdprometheus/rbac#
5，创建prometheus的deloyment,service
prometheus/prometheus#lsconfigmap.yamldeployment.yamlprometheus-rules.yamlservice.yamlprometheus/prometheus#在configmap.yaml中要注意的是在1.7以后，获取cadvsion监控pod等的信息时，用的是kubelet的4194端口，注意以下这段：这是采集cadvision信息，必须是通过kubelet的4194端口，所以Kubelet必须监听着，4194部署了cadvsion来获取pod中容器信息prometheus/prometheus#catconfigmap.yaml#https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L37-job_name:‘kubernetes-nodes‘tls_config:ca_file:/var/run/secrets/kubernetes.io/serviceaccount/ca.crtbearer_token_file:/var/run/secrets/kubernetes.io/serviceaccount/tokenkubernetes_sd_configs:-role:noderelabel_configs:-source_labels:[__address__]regex:‘(.*):10250‘replacement:‘${1}:10255‘target_label:__address__-job_name:‘kubernetes-cadvisor‘scheme:httpstls_config:ca_file:/var/run/secrets/kubernetes.io/serviceaccount/ca.crtbearer_token_file:/var/run/secrets/kubernetes.io/serviceaccount/tokenkubernetes_sd_configs:-role:noderelabel_configs:-action:labelmapregex:__meta_kubernetes_node_label_(.+)-target_label:__address__replacement:kubernetes.default.svc.cluster.local:443-source_labels:[__meta_kubernetes_node_name]regex:(.+)target_label:__metrics_path__replacement:/api/v1/nodes/${1}:4194/proxy/metrics#https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L79prometheus-rules.yaml 这是它的发现规则文件
deployment.yaml service.yaml 这两个是部署的文件， deployment部署中资源限制建议放大一点
现在部署prometheus目录下所有文件：
prometheus/prometheus#kubectlcreate-f./configmap"prometheus-core"createddeployment"prometheus-core"createdconfigmap"prometheus-rules"createdservice"prometheus"createdprometheus/prometheus#prometheus/prometheus#kubectlgetdeployment,pod-nmonitoringNAMEDESIREDCURRENTUP-TO-DATEAVAILABLEAGEdeploy/grafana-core111116mdeploy/prometheus-core11111mNAMEREADYSTATUSRESTARTSAGEpo/grafana-core-9c7f66868-wm68j1/1Running016mpo/prometheus-core-6dc6777c5b-5nc7j1/1Running01m
prometheus应用的部署，简单描述下创建的内容：
Deployment=prometheus-corereplicas:1containers=prometheusimage:prom/prometheus:v1.7.0containerPort:9090(webui)Servicename:prometheusNodePort-->port:9090-webui
6，prometheus部署完了现在来部署它的agent，也就是采集器：
Prometheus/prometheus#lsnode-directory-size-metrics/daemonset.yamlPrometheus/prometheus#lskube-state-metrics/deployment.yamlservice.yamlPrometheus/prometheus#lsnode-exporter/exporter-daemonset.yamlexporter-service.yamlPrometheus/prometheus##其中两个用的是daemonsetPrometheus/prometheus#kubectlcreate-fnode-exporter/-fkube-state-metrics/-fnode-directory-size-metrics/daemonset"prometheus-node-exporter"createdservice"prometheus-node-exporter"createddeployment"kube-state-metrics"createdservice"kube-state-metrics"createddaemonset"node-directory-size-metrics"createdPrometheus/prometheus#Prometheus/prometheus#kubectlgetdeploy,pod,svc-nmonitoringNAMEDESIREDCURRENTUP-TO-DATEAVAILABLEAGEdeploy/grafana-core111126mdeploy/kube-state-metrics22221mdeploy/prometheus-core111111mNAMEREADYSTATUSRESTARTSAGEpo/grafana-core-9c7f66868-wm68j1/1Running026mpo/kube-state-metrics-694fdcf55f-bqcp81/1Running01mpo/kube-state-metrics-694fdcf55f-nnqqd1/1Running01mpo/node-directory-size-metrics-n9wx72/2Running01mpo/node-directory-size-metrics-ppscw2/2Running01mpo/prometheus-core-6dc6777c5b-5nc7j1/1Running011mpo/prometheus-node-exporter-kchmb1/1Running01mpo/prometheus-node-exporter-lks5m1/1Running01mNAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGEsvc/grafanaNodePort10.254.231.25<none>3000:30161/TCP26msvc/kube-state-metricsClusterIP10.254.156.51<none>8080/TCP1msvc/prometheusNodePort10.254.239.90<none>9090:37318/TCP10msvc/prometheus-node-exporterClusterIPNone<none>9100/TCP1mPrometheus/prometheus#--------Prometheus/prometheus#kubectlgetpod-owide-nmonitoringNAMEREADYSTATUSRESTARTSAGEIPNODEprometheus-node-exporter-kchmb1/1Running04m10.3.1.1610.3.1.16prometheus-node-exporter-lks5m1/1Running04m10.3.1.1710.3.1.17#这两个是exporter，用的是daemonset分别在这两个node上运行了。这样就可以采集到所有数据了。
如上部署完成，以下是用自已的话简单描述下：
node-exporter/exporter-daemonset.yaml文件：DaemonSet=prometheus-node-exportercontainers：name:prometheus-node-exporterimage:prom/node-exporter:v0.14.0containerPort:9100hostPort:9100hostNetwork:true#它用的是主机的9100端口Prometheus/prometheus/node-exporter#kubectlgetdaemonset,pod-nmonitoringNAMEDESIREDCURRENTREADYUP-TO-DATEAVAILABLENODESELECTORAGEds/node-directory-size-metrics22222<none>16hds/prometheus-node-exporter22222<none>16h因为它是daemonset，所以相应的也会运行着两个Pod：prometheus-node-exporterService=prometheus-node-exporterclusterIP:Noneport:9100type:ClusterIP#它没有clusterIP#kubectlgetservice-nmonitoringNAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGEprometheus-node-exporterClusterIPNone<none>9100/TCP16h
kube-state-metrics/deployment.yaml文件：Deployment=kube-state-metricsreplicas:2containers-->name:kube-state-metricsimage:gcr.io/google_containers/kube-state-metrics:v0.5.0containerPort:8080Servicename:kube-state-metricsport:8080#没有映射#kubectlgetdeployment,pod,svc-nmonitoringNAMEDESIREDCURRENTUP-TO-DATEAVAILABLEAGEdeploy/kube-state-metrics222216hNAMEREADYSTATUSRESTARTSAGEpo/kube-state-metrics-694fdcf55f-2mmd51/1Running011hpo/kube-state-metrics-694fdcf55f-bqcp81/1Running016hNAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGEsvc/kube-state-metricsClusterIP10.254.156.51<none>8080/TCP16h
node-directory-size-metrics/daemonset.yaml文件：#因为是daemonset，所以未定义replicas数量，直接运行在每个node之上，但是它没有创建serviceDaemonSet:name:node-directory-size-metricscontainers-->name:read-duimage:giantswarm/tiny-toolsmountPath:/mnt/varmountPath:/tmpcontainers-->name:caddyimage:dockermuenster/caddy:0.9.3containerPort:9102mountPath:/var/wwwhostPath/varkubectlgetdaemonset,pod,svc-nmonitoringNAMEDESIREDCURRENTREADYUP-TO-DATEAVAILABLENODESELECTORAGEds/node-directory-size-metrics22222<none>16hNAMEREADYSTATUSRESTARTSAGEpo/node-directory-size-metrics-n9wx72/2Running016hpo/node-directory-size-metrics-ppscw2/2Running016hNAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGE没有node-directory-size-metrics的service
到此 prometheus算是部署完成了，最后来看下它暴露的端口：
Prometheus/prometheus#kubectlgetsvc-owide-nmonitoringNAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGESELECTORgrafanaNodePort10.254.231.25<none>3000:30161/TCP31mapp=grafana,component=corekube-state-metricsClusterIP10.254.156.51<none>8080/TCP6mapp=kube-state-metricsprometheusNodePort10.254.239.90<none>9090:37318/TCP16mapp=prometheus,component=coreprometheus-node-exporterClusterIPNone<none>9100/TCP6mapp=prometheus,component=node-exporterPrometheus/prometheus#
7，访问、使用prometheus
如上可以看到grafana的端口号是30161，NodeIP:30161 就可以打开grafana，默认admin/admin
登录后，添加数据源：
添加Prometheus的数据源：
将Prometheus的作为数据源的相关参数如下图所示：
添加完后，导入模板文件：
部署完成。
Kubernetes之利用prometheus监控K8S集群
原文地址：https://www.cnblogs.com/cheyunhua/p/9019927.html