继续上一章部署。
九、controller-manager部署
9.1 下载二进制文件，参考 第三章
9.2 创建 kube-controller-manager 证书和私钥
cat > kube-controller-manager-csr.json <<EOF{ ???"CN": "system:kube-controller-manager", ???"key": { ???????"algo": "rsa", ???????"size": 2048 ???}, ???"hosts": [ ?????"127.0.0.1", ?????"192.168.56.20", ?????"192.168.56.21" ???], ???"names": [ ?????{ ???????"C": "CN", ???????"ST": "BeiJing", ???????"L": "BeiJing", ???????"O": "system:kube-controller-manager", ???????"OU": "4Paradigm" ?????} ???]}EOF- hosts 列表包含所有 kube-controller-manager 节点 IP
- CN 为 system:kube-controller-manager、O 为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限
生成证书和私钥：
cfssl gencert -ca=/etc/kubernetes/cert/ca.pem ??-ca-key=/etc/kubernetes/cert/ca-key.pem ??-config=/etc/kubernetes/cert/ca-config.json ??-profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-managerls kube-controller-manager*.pem
分发证书到其他master节点：
source /opt/k8s/bin/environment.shfor master_ip in ${MASTER_IPS[@]} ?do ???echo ">>> ${master_ip}" ???scp kube-controller-manager*.pem k8s@${master_ip}:/etc/kubernetes/cert/done9.3 创建和分发 kubeconfig 文件
source /opt/k8s/bin/environment.shkubectl config set-cluster kubernetes ??--certificate-authority=/etc/kubernetes/cert/ca.pem ??--embed-certs=true ??--server=${KUBE_APISERVER} ??--kubeconfig=kube-controller-manager.kubeconfigkubectl config set-credentials system:kube-controller-manager ??--client-certificate=kube-controller-manager.pem ??--client-key=kube-controller-manager-key.pem ??--embed-certs=true ??--kubeconfig=kube-controller-manager.kubeconfigkubectl config set-context system:kube-controller-manager ??--cluster=kubernetes ??--user=system:kube-controller-manager ??--kubeconfig=kube-controller-manager.kubeconfigkubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig分发 kubeconfig 文件到所有 master 节点：
source /opt/k8s/bin/environment.shfor master_ip in ${MASTER_IPS[@]} ?do ???echo ">>> ${master_ip}" ???scp kube-controller-manager.kubeconfig k8s@${master_ip}:/etc/kubernetes/done9.4 创建和分发 kube-controller-manager systemd unit 文件
source /opt/k8s/bin/environment.shcat > kube-controller-manager.service <<EOF[Unit]Description=Kubernetes Controller ManagerDocumentation=https://github.com/GoogleCloudPlatform/kubernetes[Service]ExecStart=/opt/k8s/bin/kube-controller-manager \ ?--port=0 \ ?--secure-port=10252 \ ?--bind-address=127.0.0.1 \ ?--kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \ ?--service-cluster-ip-range=${SERVICE_CIDR} \ ?--cluster-name=kubernetes \ ?--cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \ ?--cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \ ?--experimental-cluster-signing-duration=8760h \ ?--root-ca-file=/etc/kubernetes/cert/ca.pem \ ?--service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \ ?--leader-elect=true \ ?--feature-gates=RotateKubeletServerCertificate=true \ ?--controllers=*,bootstrapsigner,tokencleaner \ ?--horizontal-pod-autoscaler-use-rest-clients=true \ ?--horizontal-pod-autoscaler-sync-period=10s \ ?--tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \ ?--tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \ ?--use-service-account-credentials=true \ ?--alsologtostderr=true \ ?--logtostderr=false \ ?--log-dir=/var/log/kubernetes \ ?--v=2Restart=onRestart=on-failureRestartSec=5User=k8s[Install]WantedBy=multi-user.targetEOF- --port=0：关闭监听 http /metrics 的请求，同时 --address 参数无效，--bind-address 参数有效
- --secure-port=10252、--bind-address=0.0.0.0: 在所有网络接口监听 10252 端口的 https /metrics 请求
- --kubeconfig：指定 kubeconfig 文件路径，kube-controller-manager 使用它连接和验证 kube-apiserver
- --cluster-signing-*-file：签名 TLS Bootstrap 创建的证书
- --experimental-cluster-signing-duration：指定 TLS Bootstrap 证书的有效期
- --root-ca-file：放置到容器 ServiceAccount 中的 CA 证书，用来对 kube-apiserver 的证书进行校验
- --service-account-private-key-file：签名 ServiceAccount 中 Token 的私钥文件，必须和 kube-apiserver 的 --service-account-key-file 指定的公钥文件配对使用
- --service-cluster-ip-range ：指定 Service Cluster IP 网段，必须和 kube-apiserver 中的同名参数一致
- --leader-elect=true：集群运行模式，启用选举功能被选为 leader 的节点负责处理工作，其它节点为阻塞状态
- --feature-gates=RotateKubeletServerCertificate=true：开启 kublet server 证书的自动更新特性
- --controllers=*,bootstrapsigner,tokencleaner：启用的控制器列表，tokencleaner 用于自动清理过期的 Bootstrap token
- --horizontal-pod-autoscaler-*：custom metrics 相关参数，支持 autoscaling/v2alpha1
- --tls-cert-file、--tls-private-key-file：使用 https 输出 metrics 时使用的 Server 证书和秘钥
- --use-service-account-credentials=true:
- User=k8s：使用 k8s 账户运行
kube-controller-manager 不对请求 https metrics 的 Client 证书进行校验，故不需要指定 --tls-ca-file 参数，而且该参数已被淘汰
分发 systemd unit 文件到所有master节点：
source /opt/k8s/bin/environment.shfor master_ip in ${MASTER_IPS[@]} ?do ???echo ">>> ${master_ip}" ???scp kube-controller-manager.service root@${master_ip}:/etc/systemd/system/donekube-controller-manager 的权限
ClusteRole: system:kube-controller-manager 的权限很小，只能创建 secret、serviceaccount 等资源对象，各 controller 的权限分散到 ClusterRole system:controller:XXX 中。
需要在 kube-controller-manager 的启动参数中添加 --use-service-account-credentials=true 参数，这样 main controller 会为各 controller 创建对应的 ServiceAccount XXX-controller。
内置的 ClusterRoleBinding system:controller:XXX 将赋予各 XXX-controller ServiceAccount 对应的 ClusterRole system:controller:XXX 权限。
9.5 启动 kube-controller-manager 服务
source /opt/k8s/bin/environment.shfor master_ip in ${MASTER_IPS[@]} ?do ???echo ">>> ${master_ip}" ???ssh root@${master_ip} "mkdir -p /var/log/kubernetes && chown -R k8s /var/log/kubernetes" ???ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager"done检查 kube-controller-manager 服务状态
source /opt/k8s/bin/environment.shfor master_ip in ${MASTER_IPS[@]} ?do ???echo ">>> ${master_ip}" ???ssh k8s@${master_ip} "systemctl status kube-controller-manager|grep Active"done确保都是 Active: active (running) 状态
查看日志：journalctl -u kube-controller-manager
9.6 查看输出的 Metric
注：下面的命令在 kube-controller-manager 上执行
kube-controller-manager 监听10252端口，接收 https 请求
[root@k8s-m1 template]# sudo netstat -lnpt|grep kube-controlltcp ???????0 ?????0 127.0.0.1:10252 ????????0.0.0.0:* ??????????????LISTEN ?????23543/kube-controll [root@k8s-m1 template]# curl -s --cacert /etc/kubernetes/cert/ca.pem https://127.0.0.1:10252/metrics |head# HELP ClusterRoleAggregator_adds Total number of adds handled by workqueue: ClusterRoleAggregator# TYPE ClusterRoleAggregator_adds counterClusterRoleAggregator_adds 7# HELP ClusterRoleAggregator_depth Current depth of workqueue: ClusterRoleAggregator# TYPE ClusterRoleAggregator_depth gaugeClusterRoleAggregator_depth 0# HELP ClusterRoleAggregator_queue_latency How long an item stays in workqueueClusterRoleAggregator before being requested.# TYPE ClusterRoleAggregator_queue_latency summaryClusterRoleAggregator_queue_latency{quantile="0.5"} NaNClusterRoleAggregator_queue_latency{quantile="0.9"} NaN- curl --cacert CA 证书用来验证 kube-controller-manager https server 证书
9.7 测试 kube-controller-manager 高可用
停掉一个节点的 kube-controller-manager 服务，观察其他节点的日志，看是否获取了 leader 权限
查看当前的 leader：
[root@k8s-m1 template]# kubectl get endpoints kube-controller-manager --namespace=kube-system ?-o yamlapiVersion: v1kind: Endpointsmetadata: ?annotations: ???control-plane.alpha.kubernetes.io/leader: ‘{"holderIdentity":"k8s-m1_1a33e4f7-d83a-11e8-92ec-080027675bc1","leaseDurationSeconds":15,"acquireTime":"2018-10-25T09:41:09Z","renewTime":"2018-10-25T10:09:32Z","leaderTransitions":0}‘ ?creationTimestamp: 2018-10-25T09:41:09Z ?name: kube-controller-manager ?namespace: kube-system ?resourceVersion: "4022" ?selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager ?uid: 1a397e61-d83a-11e8-a1d2-080027675bc1当前 leader 是 k8s-m1
参考：
- 关于 controller 权限和 use-service-account-credentials 参数：https://github.com/kubernetes/kubernetes/issues/48208
- kublet 认证和授权：https://kubernetes.io/docs/admin/kubelet-authentication-authorization/#kubelet-authorization
二进制安装kubernetes v1.11.2 （第九章 controller-manager部署）
原文地址：https://www.cnblogs.com/aast/p/9851533.html