其实现原理有点类似ELK、EFK组合。node-exporter组件负责收集节点上的metrics监控数据，并将数据推送给prometheus, prometheus负责存储这些数据，grafana将这些数据通过网页以图形的形式展现给用户。
在开始之前有必要了解下Prometheus是什么?
Prometheus （中文名：普罗米修斯）是由 SoundCloud 开发的开源监控报警系统和时序列数据库(TSDB).自2012年起,许多公司及组织已经采用 Prometheus,并且该项目有着非常活跃的开发者和用户社区.现在已经成为一个独立的开源项目。Prometheus 在2016加入 CNCF ( Cloud Native Computing Foundation ), 作为在 kubernetes 之后的第二个由基金会主持的项目。 Prometheus 的实现参考了Google内部的监控实现，与源自Google的Kubernetes结合起来非常合适。另外相比influxdb的方案，性能更加突出，而且还内置了报警功能。它针对大规模的集群环境设计了拉取式的数据采集方式，只需要在应用里面实现一个metrics接口，然后把这个接口告诉Prometheus就可以完成数据采集了，下图为prometheus的架构图。
Prometheus的特点：
1、多维数据模型（时序列数据由metric名和一组key/value组成）
2、在多维度上灵活的查询语言(PromQl)
3、不依赖分布式存储，单主节点工作.
4、通过基于HTTP的pull方式采集时序数据
5、可以通过中间网关进行时序列数据推送(pushing)
6、目标服务器可以通过发现服务或者静态配置实现
7、多种可视化和仪表盘支持
prometheus 相关组件，Prometheus生态系统由多个组件组成，其中许多是可选的：
1、Prometheus 主服务,用来抓取和存储时序数据
2、client library 用来构造应用或 exporter 代码 (go,java,python,ruby)
3、push 网关可用来支持短连接任务
4、可视化的dashboard (两种选择,promdash 和 grafana.目前主流选择是 grafana.)
4、一些特殊需求的数据出口(用于HAProxy, StatsD, Graphite等服务)
5、实验性的报警管理端(alartmanager,单独进行报警汇总,分发,屏蔽等 )
promethues 的各个组件基本都是用 golang 编写,对编译和部署十分友好.并且没有特殊依赖.基本都是独立工作。
上述文字来自网络！
现在我们正式开始部署工作。
一、环境介绍
操作系统环境：centos linux 7.2 64bit
K8S软件版本： 1.9.0（采用kubeadm方式部署）
Master节点IP： 192.168.115.5/24
Node节点IP： ??192.168.115.6/24
二、在k8s集群的所有节点上下载所需要的image
# docker pull prom/node-exporter# docker pull prom/prometheus:v2.0.0# docker pull grafana/grafana:4.2.0三、采用daemonset方式部署node-exporter组件
# cat node-exporter.yaml ---apiVersion: extensions/v1beta1kind: DaemonSetmetadata: ?name: node-exporter ?namespace: kube-system ?labels: ???k8s-app: node-exporterspec: ?template: ???metadata: ?????labels: ???????k8s-app: node-exporter ???spec: ?????containers: ?????- image: prom/node-exporter ???????name: node-exporter ???????ports: ???????- containerPort: 9100 ?????????protocol: TCP ?????????name: http---apiVersion: v1kind: Servicemetadata: ?labels: ???k8s-app: node-exporter ?name: node-exporter ?namespace: kube-systemspec: ?ports: ?- name: http ???port: 9100 ???nodePort: 31672 ???protocol: TCP ?type: NodePort ?selector: ???k8s-app: node-exporter通过上述文件创建pod和service
# kubectl create -f ?node-exporter.yaml 四、部署prometheus组件
1、rbac文件
# cat rbac-setup.yaml apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: ?name: prometheusrules:- apiGroups: [""] ?resources: ?- nodes ?- nodes/proxy ?- services ?- endpoints ?- pods ?verbs: ["get", "list", "watch"]- apiGroups: ?- extensions ?resources: ?- ingresses ?verbs: ["get", "list", "watch"]- nonResourceURLs: ["/metrics"] ?verbs: ["get"]---apiVersion: v1kind: ServiceAccountmetadata: ?name: prometheus ?namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: ?name: prometheusroleRef: ?apiGroup: rbac.authorization.k8s.io ?kind: ClusterRole ?name: prometheussubjects:- kind: ServiceAccount ?name: prometheus ?namespace: kube-system2、以configmap的形式管理prometheus组件的配置文件
# cat configmap.yaml apiVersion: v1kind: ConfigMapmetadata: ?name: prometheus-config ?namespace: kube-systemdata: ?prometheus.yml: | ???global: ?????scrape_interval: ????15s ?????evaluation_interval: 15s ???scrape_configs: ???- job_name: ‘kubernetes-apiservers‘ ?????kubernetes_sd_configs: ?????- role: endpoints ?????scheme: https ?????tls_config: ???????ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt ?????bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token ?????relabel_configs: ?????- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] ???????action: keep ???????regex: default;kubernetes;https ???- job_name: ‘kubernetes-nodes‘ ?????kubernetes_sd_configs: ?????- role: node ?????scheme: https ?????tls_config: ???????ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt ?????bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token ?????relabel_configs: ?????- action: labelmap ???????regex: __meta_kubernetes_node_label_(.+) ?????- target_label: __address__ ???????replacement: kubernetes.default.svc:443 ?????- source_labels: [__meta_kubernetes_node_name] ???????regex: (.+) ???????target_label: __metrics_path__ ???????replacement: /api/v1/nodes/${1}/proxy/metrics ???- job_name: ‘kubernetes-cadvisor‘ ?????kubernetes_sd_configs: ?????- role: node ?????scheme: https ?????tls_config: ???????ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt ?????bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token ?????relabel_configs: ?????- action: labelmap ???????regex: __meta_kubernetes_node_label_(.+) ?????- target_label: __address__ ???????replacement: kubernetes.default.svc:443 ?????- source_labels: [__meta_kubernetes_node_name] ???????regex: (.+) ???????target_label: __metrics_path__ ???????replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor ???- job_name: ‘kubernetes-service-endpoints‘ ?????kubernetes_sd_configs: ?????- role: endpoints ?????relabel_configs: ?????- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] ???????action: keep ???????regex: true ?????- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] ???????action: replace ???????target_label: __scheme__ ???????regex: (https?) ?????- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] ???????action: replace ???????target_label: __metrics_path__ ???????regex: (.+) ?????- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] ???????action: replace ???????target_label: __address__ ???????regex: ([^:]+)(?::\d+)?;(\d+) ???????replacement: $1:$2 ?????- action: labelmap ???????regex: __meta_kubernetes_service_label_(.+) ?????- source_labels: [__meta_kubernetes_namespace] ???????action: replace ???????target_label: kubernetes_namespace ?????- source_labels: [__meta_kubernetes_service_name] ???????action: replace ???????target_label: kubernetes_name ???- job_name: ‘kubernetes-services‘ ?????kubernetes_sd_configs: ?????- role: service ?????metrics_path: /probe ?????params: ???????module: [http_2xx] ?????relabel_configs: ?????- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe] ???????action: keep ???????regex: true ?????- source_labels: [__address__] ???????target_label: __param_target ?????- target_label: __address__ ???????replacement: blackbox-exporter.example.com:9115 ?????- source_labels: [__param_target] ???????target_label: instance ?????- action: labelmap ???????regex: __meta_kubernetes_service_label_(.+) ?????- source_labels: [__meta_kubernetes_namespace] ???????target_label: kubernetes_namespace ?????- source_labels: [__meta_kubernetes_service_name] ???????target_label: kubernetes_name ???- job_name: ‘kubernetes-ingresses‘ ?????kubernetes_sd_configs: ?????- role: ingress ?????relabel_configs: ?????- source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe] ???????action: keep ???????regex: true ?????- source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path] ???????regex: (.+);(.+);(.+) ???????replacement: ${1}://${2}${3} ???????target_label: __param_target ?????- target_label: __address__ ???????replacement: blackbox-exporter.example.com:9115 ?????- source_labels: [__param_target] ???????target_label: instance ?????- action: labelmap ???????regex: __meta_kubernetes_ingress_label_(.+) ?????- source_labels: [__meta_kubernetes_namespace] ???????target_label: kubernetes_namespace ?????- source_labels: [__meta_kubernetes_ingress_name] ???????target_label: kubernetes_name ???- job_name: ‘kubernetes-pods‘ ?????kubernetes_sd_configs: ?????- role: pod ?????relabel_configs: ?????- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] ???????action: keep ???????regex: true ?????- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] ???????action: replace ???????target_label: __metrics_path__ ???????regex: (.+) ?????- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] ???????action: replace ???????regex: ([^:]+)(?::\d+)?;(\d+) ???????replacement: $1:$2 ???????target_label: __address__ ?????- action: labelmap ???????regex: __meta_kubernetes_pod_label_(.+) ?????- source_labels: [__meta_kubernetes_namespace] ???????action: replace ???????target_label: kubernetes_namespace ?????- source_labels: [__meta_kubernetes_pod_name] ???????action: replace ???????target_label: kubernetes_pod_name3、Prometheus deployment 文件
# cat prometheus.deploy.yml ---apiVersion: apps/v1beta2kind: Deploymentmetadata: ?labels: ???name: prometheus-deployment ?name: prometheus ?namespace: kube-systemspec: ?replicas: 1 ?selector: ???matchLabels: ?????app: prometheus ?template: ???metadata: ?????labels: ???????app: prometheus ???spec: ?????containers: ?????- image: prom/prometheus:v2.0.0 ???????name: prometheus ???????command: ???????- "/bin/prometheus" ???????args: ???????- "--config.file=/etc/prometheus/prometheus.yml" ???????- "--storage.tsdb.path=/prometheus" ???????- "--storage.tsdb.retention=24h" ???????ports: ???????- containerPort: 9090 ?????????protocol: TCP ???????volumeMounts: ???????- mountPath: "/prometheus" ?????????name: data ???????- mountPath: "/etc/prometheus" ?????????name: config-volume ???????resources: ?????????requests: ???????????cpu: 100m ???????????memory: 100Mi ?????????limits: ???????????cpu: 500m ???????????memory: 2500Mi ?????serviceAccountName: prometheus ?????????volumes: ?????- name: data ???????emptyDir: {} ?????- name: config-volume ???????configMap: ?????????name: prometheus-config ??????4、Prometheus service文件
# cat prometheus.svc.yml ---kind: ServiceapiVersion: v1metadata: ?labels: ???app: prometheus ?name: prometheus ?namespace: kube-systemspec: ?type: NodePort ?ports: ?- port: 9090 ???targetPort: 9090 ???nodePort: 30003 ?selector:app: prometheus5、通过上述yaml文件创建相应的对象
# kubectl create -f ?rbac-setup.yaml# kubectl create -f ?configmap.yaml # kubectl create -f ?prometheus.deploy.yml # kubectl create -f ?prometheus.svc.yml
Node-exporter对应的nodeport端口为31672，通过访问http://192.168.115.5:31672/metrics 可以看到对应的metrics
prometheus对应的nodeport端口为30003，通过访问http://192.168.115.5:30003/target 可以看到prometheus已经成功连接上了k8s的apiserver
可以在prometheus的WEB界面上提供了基本的查询K8S集群中每个POD的CPU使用情况，查询条件如下：
sum by (pod_name)( rate(container_cpu_usage_seconds_total{image!="", pod_name!=""}[1m] ) )
上述的查询有出现数据，说明node-exporter往prometheus中写入数据正常，接下来我们就可以部署grafana组件，实现更友好的webui展示数据了。
五、部署grafana组件
1、grafana deployment配置文件
# cat grafana-deploy.yaml apiVersion: extensions/v1beta1kind: Deploymentmetadata: ?name: grafana-core ?namespace: kube-system ?labels: ???app: grafana ???component: corespec: ?replicas: 1 ?template: ???metadata: ?????labels: ???????app: grafana ???????component: core ???spec: ?????containers: ?????- image: grafana/grafana:4.2.0 ???????name: grafana-core ???????imagePullPolicy: IfNotPresent ???????# env: ???????resources: ?????????# keep request = limit to keep this container in guaranteed class ?????????limits: ???????????cpu: 100m ???????????memory: 100Mi ?????????requests: ???????????cpu: 100m ???????????memory: 100Mi ???????env: ?????????# The following env variables set up basic auth twith the default admin user and admin password. ?????????- name: GF_AUTH_BASIC_ENABLED ???????????value: "true" ?????????- name: GF_AUTH_ANONYMOUS_ENABLED ???????????value: "false" ?????????# - name: GF_AUTH_ANONYMOUS_ORG_ROLE ?????????# ??value: Admin ?????????# does not really work, because of template variables in exported dashboards: ?????????# - name: GF_DASHBOARDS_JSON_ENABLED ?????????# ??value: "true" ???????readinessProbe: ?????????httpGet: ???????????path: /login ???????????port: 3000 ?????????# initialDelaySeconds: 30 ?????????# timeoutSeconds: 1 ???????volumeMounts: ???????- name: grafana-persistent-storage ?????????mountPath: /var ?????volumes: ?????- name: grafana-persistent-storage ???????emptyDir: {}2、grafana service配置文件
# cat grafana-svc.yaml apiVersion: v1kind: Servicemetadata: ?name: grafana ?namespace: kube-system ?labels: ???app: grafana ???component: corespec: ?type: NodePort ?ports: ???- port: 3000 ?selector: ???app: grafanacomponent: core3、grafana ingress配置文件# cat grafana-ing.yaml apiVersion: extensions/v1beta1kind: Ingressmetadata: ??name: grafana ??namespace: kube-systemspec: ??rules: ??- host: k8s.grafana ????http: ??????paths: ??????- path: / ????????backend: ?????????serviceName: grafana ?????????servicePort: 3000通过访问traefik的webui可以看到k8s.grafana服务发布成功
修改hosts解析，访问测试
也可以直接访问nodeport端口
默认用户名和密码都是admin
配置数据源为prometheus
导入面板，可以直接输入模板编号315在线导入，或者下载好对应的json模板文件本地导入，面板模板下载地址https://grafana.com/dashboards/315
导入面板之后就可以看到对应的监控数据了。
这里要说明一下，在测试过程中，导入编号为162的模板，发现只有部分数据，且pod的名称显示不友好。模板地址https://grafana.com/dashboards/162，详见下图。
六、后记
这里存在一些问题后续要继续研究解决。
1、prometheus的数据存储采用emptydir。如果Pod被删除，或者Pod发生迁移，emptyDir也会被删除，并且永久丢失。后续可以在K8S集群外部再配置一个Prometheus系统来永久保存监控数据， 两个prometheus系统之间通过配置job自动进行数据拉取。
2、Grafana的配置数据存储采用emptydir。如果Pod被删除，或者Pod发生迁移，emptyDir也会被删除，并且永久丢失。我们也可以选择将grafana配置在k8s外部，数据源选择K8S集群外部的prometheus即可。
3、关于监控项的报警（alertmanager）尚未配置。
参考文档，感谢作者分享！
https://www.kubernetes.org.cn/3418.html
https://blog.qikqiak.com/post/kubernetes-monitor-prometheus-grafana/
https://github.com/giantswarm/kubernetes-prometheus/tree/master/manifests
https://segmentfault.com/a/1190000013245394
Kubernetes集群监控方案
原文地址：http://blog.51cto.com/ylw6006/2084403