继续部署。
七、部署flannel网络插件
kubernetes要求集群内各节点（包括master节点）能通过pod网段互联互通。flannel使用vxlan技术为各节点创建一个可以互通的pod网络，使用的端口为UDP 8472，需要开放该端口（如公有云aws等）。
flannel第一次启动时，从etcd获取pod网段信息，为本节点分配 一个未使用的/24段地址，然后创建flannel开头的一个接口。
flannel将分配的pod网段信息写入/run/flannel/docker文件，docker后续使用这个文件中的环境变量设置docker0网桥。
7.1 下载二进制文件
下载地址：https://github.com/coreos/flannel/releases
# 下载二进制文件[k8s@k8s-m1 ~]$ mkdir -p /home/k8s/k8s/flannel-v0.10.0 && cd /home/k8s/k8s[k8s@k8s-m1 k8s]$ wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz[k8s@k8s-m1 k8s]$ tar -xzvf flannel-v0.10.0-linux-amd64.tar.gz -C flannel-v0.10.0# 分发到所有节点[k8s@k8s-m1 k8s]$ source /opt/k8s/bin/environment.sh[k8s@k8s-m1 k8s]$ for node_ip in ${NODE_IPS[@]} ?do ???echo ">>> ${node_ip}" ???scp ?flannel-v0.10.0/{flanneld,mk-docker-opts.sh} k8s@${node_ip}:/opt/k8s/bin/ ???ssh k8s@${node_ip} "chmod +x /opt/k8s/bin/*"done7.2 创建flannel证书和私钥
flannel从etcd集群存取网段分配信息，而etcd集群启用了双向x509证书认证，所以需要为flannel生成证书和私钥。
# 创建证书签名请求[k8s@k8s-m1 cert]$ cd /opt/k8s/cert[k8s@k8s-m1 cert]$ cat > flanneld-csr.json <<EOF{ ?"CN": "flanneld", ?"hosts": [], ?"key": { ???"algo": "rsa", ???"size": 2048 ?}, ?"names": [ ???{ ?????"C": "CN", ?????"ST": "BeiJing", ?????"L": "BeiJing", ?????"O": "k8s", ?????"OU": "4Paradigm" ???} ?]}EOF# 生成证书和私钥[k8s@k8s-m1 cert]$ cfssl gencert -ca=/etc/kubernetes/cert/ca.pem ??-ca-key=/etc/kubernetes/cert/ca-key.pem ??-config=/etc/kubernetes/cert/ca-config.json ??-profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld[k8s@k8s-m1 cert]$ ls flanneld*pem# 分发证书和私钥到各节点[k8s@k8s-m1 cert]$ for node_ip in ${NODE_IPS[@]} ?do ???echo ">>> ${node_ip}" ???ssh root@${node_ip} "mkdir -p /etc/flanneld/cert && chown -R k8s /etc/flanneld" ???scp flanneld*.pem k8s@${node_ip}:/etc/flanneld/certdone7.2 向etcd写入集群pod网段信息
source /opt/k8s/bin/environment.shetcdctl ??--endpoints=${ETCD_ENDPOINTS} ??--ca-file=/etc/kubernetes/cert/ca.pem ??--cert-file=/etc/flanneld/cert/flanneld.pem ??--key-file=/etc/flanneld/cert/flanneld-key.pem ??set ${FLANNEL_ETCD_PREFIX}/config ‘{"Network":"‘${CLUSTER_CIDR}‘", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}‘- flannel当前版本（v0.10.0）不支持etcd v3，所以使用etcd v2 API写入配置key和网段数据；
- 写入的pod网段${CLUSTER_CIDR}必须是/16段地址必须与kube-controller-manager的--cluster-cidr参数值一致；
7.3 创建flanneld的system unit文件
source /opt/k8s/bin/environment.shexport IFACE=eth0cat > flanneld.service << EOF[Unit]Description=Flanneld overlay address etcd agentAfter=network.targetAfter=network-online.targetWants=network-online.targetAfter=etcd.serviceBefore=docker.service[Service]Type=notifyExecStart=/opt/k8s/bin/flanneld \ ?-etcd-cafile=/etc/kubernetes/cert/ca.pem \ ?-etcd-certfile=/etc/flanneld/cert/flanneld.pem \ ?-etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \ ?-etcd-endpoints=${ETCD_ENDPOINTS} \ ?-etcd-prefix=${FLANNEL_ETCD_PREFIX} \ ?-iface=${IFACE}ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/dockerRestart=on-failure[Install]WantedBy=multi-user.targetRequiredBy=docker.serviceEOF- mk-docker-opts.sh脚本将分配给flanneld的pod子网网段信息写入/run/flannel/docker文件，后续docker启动时使用这个文件中的环境变量配置docker0网桥
- flanneld使用系统默认路由所在的接口与其他节点通信，对于有多个网络接口（如内网和公网）的节点，可以用-iface参数指定通信接口，如上面的enp0s3接口
- flanneld运行时需要root权限
完整的unit文件如下
[k8s@k8s-m1 template]$ cat flanneld.service [Unit]Description=Flanneld overlay address etcd agentAfter=network.targetAfter=network-online.targetWants=network-online.targetAfter=etcd.serviceBefore=docker.service[Service]Type=notifyExecStart=/opt/k8s/bin/flanneld ??-etcd-cafile=/etc/kubernetes/cert/ca.pem ??-etcd-certfile=/etc/flanneld/cert/flanneld.pem ??-etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem ??-etcd-endpoints=https://192.168.56.20:2379,https://192.168.56.21:2379 \ ?-etcd-prefix=/kubernetes/network ??-iface=ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/dockerRestart=on-failure[Install]WantedBy=multi-user.targetRequiredBy=docker.service
7.4 分发flanneld systemd unit文件到所有节点
source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]} ?do ???echo ">>> ${node_ip}" ???scp flanneld.service root@${node_ip}:/etc/systemd/system/done7.5 启动flanneld服务
source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]} ?do ???echo ">>> ${node_ip}" ???ssh root@${node_ip} "systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld"done7.6 检查分配给各flanneld的pod网段信息
查看集群pod网段（/16）：
source /opt/k8s/bin/environment.shetcdctl ??--endpoints=${ETCD_ENDPOINTS} ??--ca-file=/etc/kubernetes/cert/ca.pem ??--cert-file=/etc/flanneld/cert/flanneld.pem ??--key-file=/etc/flanneld/cert/flanneld-key.pem ??get ${FLANNEL_ETCD_PREFIX}/config输出：
{"Network":"172.30.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}查看已经分配的pod子网列表（/24）：
source /opt/k8s/bin/environment.shetcdctl ??--endpoints=${ETCD_ENDPOINTS} ??--ca-file=/etc/kubernetes/cert/ca.pem ??--cert-file=/etc/flanneld/cert/flanneld.pem ??--key-file=/etc/flanneld/cert/flanneld-key.pem ??ls ${FLANNEL_ETCD_PREFIX}/subnets输出：
/kubernetes/network/subnets/172.30.17.0-24/kubernetes/network/subnets/172.30.5.0-24/kubernetes/network/subnets/172.30.75.0-24/kubernetes/network/subnets/172.30.57.0-24
查看某一pod网段对应节点IP和flannel接口地址：
source /opt/k8s/bin/environment.shetcdctl ??--endpoints=${ETCD_ENDPOINTS} ??--ca-file=/etc/kubernetes/cert/ca.pem ??--cert-file=/etc/flanneld/cert/flanneld.pem ??--key-file=/etc/flanneld/cert/flanneld-key.pem ??get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.17.0-24输出：
{"PublicIP":"192.168.72.68","BackendType":"vxlan","BackendData":{"VtepMAC":"c2:d6:2a:e4:96:ce"}}7.7 验证各节点能通过pod网段互通
在各节点部署flannel后，检查是否创建了flannel接口
source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]} ?do ???echo ">>> ${node_ip}" ???ssh ${node_ip} "/usr/sbin/ip addr show flannel.1|grep -w inet"done输出：
>>> 192.168.56.20 ???inet 172.30.57.0/32 scope global flannel.1>>> 192.168.56.21 ???inet 172.30.17.0/32 scope global flannel.1>>> 192.168.56.30 ???inet 172.30.5.0/32 scope global flannel.1>>> 192.168.56.31 ???inet 172.30.75.0/32 scope global flannel.1
在各节点上ping所有flannel接口ip，确保能通：
source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]} ?do ???echo ">>> ${node_ip}" ???ssh ${node_ip} "ping -c 1 172.30.57.0" ???ssh ${node_ip} "ping -c 1 172.30.17.0" ???ssh ${node_ip} "ping -c 1 172.30.5.0" ???ssh ${node_ip} "ping -c 1 172.30.75.0"done二进制安装kubernetes v1.11.2 （第五章 flannel网络插件部署）
原文地址：https://www.cnblogs.com/aast/p/9842626.html