苏打

kubeadm搭建高可用集群

2019-11-18 · 6 min read
kubernetes kubeadm

使用 kubeadm 搭建单节点集群已经介绍过了,关于如何初始化环境,安装依赖组件请参考kubeadm搭建单节点集群, 这篇文章介绍如何使用kubeadm搭建高可用集群。

高可用拓扑模型

堆叠ETCD模型(Stacked etcd topology)

该模式主要特点:

  1. etcd和k8s master 组件部署在同一个node上
  2. 每个kube-apiserver、kube-schedule、kube-controller-manager 只连接本地etcd服务

{% asset_img kubeadm-ha-topology-stacked-etcd.svg %}

外部ETCD模型(External etcd topology)

该模式主要特点:

  1. etcd集群单独部署,不通过kubeadm创建管理
  2. 每个kube-apiserver、kube-schedule、kube-controller-manager 会连接到外部etcd集群

{% asset_img kubeadm-ha-topology-external-etcd.svg %}

使用kubeadm创建集群时,如果etcd和master部署在同一个节点时使用 堆叠ETCD模型(Stacked etcd topology),否则配置使用外部ETCD模型(External etcd topology)

创建 kube-api 负载均衡

如以上集群结构图可知,我们首先需要为多个master节点创建负载均衡,负载均衡的配置需要注意以下几点:

  • 负载均衡创建成功后,需要将 kubaadm 的 ControlPlaneEndpoint 字段配置为负载均衡ip
  • 需要先把第一个待初始化master节点挂载到负载均衡后端,当第一个master创建完成后,将下一个初始化的master节点ip加入到负载均衡后端,以此类推,直到所有master节点创建完毕。

创建高可用集群

1. 配置kubeadm config 文件

  • 创建堆叠ETCD模型的配置:
cat <<EOF > /root/kubeadm-config
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.13.4
apiServer:
  certSANs:
  - "MASTER1_IP"
  - "MASTER2_IP"
  - "MASTER3_IP"
  - "LOAD_BALANCER_IP"
controlPlaneEndpoint: "LOAD_BALANCER_IP:LOAD_BALANCER_PORT"
imageRepository: registry.cn-beijing.aliyuncs.com/hsxue
networking:
  dnsDomain: cluster.local
  podSubnet: 10.128.0.0/23
  serviceSubnet: 10.192.0.0/22
EOF

配置中的 MASTER1_IP、MASTER2_IP、 MASTER3_IP、LOAD_BALANCER_IP、LOAD_BALANCER_PORT 需要替换成实际的值

  • 创建外部ETCD模型的配置
cat <<EOF > /root/kubeadm-config
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.13.4
apiServer:
  certSANs:
  - "MASTER1_IP"
  - "MASTER2_IP"
  - "MASTER3_IP"
  - "LOAD_BALANCER_IP"
controlPlaneEndpoint: "LOAD_BALANCER_IP:LOAD_BALANCER_PORT"
imageRepository: registry.cn-beijing.aliyuncs.com/hsxue
networking:
  dnsDomain: cluster.local
  podSubnet: 10.128.0.0/23
  serviceSubnet: 10.192.0.0/22
etcd:
    external:
        endpoints:
        - https://ETCD_0_IP:2379
        - https://ETCD_1_IP:2379
        - https://ETCD_2_IP:2379
        caFile: /etc/kubernetes/pki/etcd/ca.crt
        certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
        keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key

如上配置增加了外部etcd的配置,配置中如下值需要替换成实际的值:

  • MASTER1_IP
  • MASTER2_IP
  • MASTER3_IP
  • LOAD_BALANCER_IP
  • LOAD_BALANCER_PORT
  • ETCD_0_IP
  • ETCD_1_IP
  • ETCD_2_IP

2. 初始化第一个master节点

如果安装的是外部ETCD模型集群,需要先将etcd证书拷贝到当前master节点,否则不需要执行这一步

export CONTROL_PLANE="MASTER1_IP"
scp /etc/kubernetes/pki/etcd/ca.crt "${CONTROL_PLANE}":
scp /etc/kubernetes/pki/apiserver-etcd-client.crt "${CONTROL_PLANE}":
scp /etc/kubernetes/pki/apiserver-etcd-client.key "${CONTROL_PLANE}":

注意:etcd证书需要和上一步配置的etcd证书路径匹配


将该节点kube-apiserver地址和端口挂载到负载均衡后端,然后执行如下命令

kubeadm init --config=/root/kubeadm-config --node-name=master1 --upload-certs

部署成功后会输出如下信息:

...
You can now join any number of control-plane node by running the following command on each as a root:
kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866 --control-plane --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use kubeadm init phase upload-certs to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866

以上信息有两行关键信息需要记下来(每个人安装完成后,输出的信息肯定不一样):

  1. 加入master节点命令
kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866 --control-plane --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07
  1. 加入worker节点命令
kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866

3. 安装pod网络插件

如下安装 flannel 为例,执行如下命令:

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

执行 kubectl get pod -n kube-system -w 等待master组件全部 running

详情参考:pod network addon

4. 加入剩余的master节点到集群

先将待初始化节点kube-apiserver地址和端口挂载到负载均衡后端,然后执行上面得到的加入master节点的命令

kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866 --control-plane --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07
  • --control-plane 参数表示 kubeadm join 命令创建的是一个master节点
  • --certificate-key master加入集群时,会从 kubeadm-certs Secret下载证书,并使用该certificate-key解密证书,
kubeadm-certs Secret 和 certificate-key 2小时后会过期,如果过期则需要重新生成

重新生成certificate-key,可执行如下命令

kubeadm init phase upload-certs --upload-certs

执行完成后,继续登陆到剩余master节点做同样操作。直到所有master节点创建完毕

5. 加入worker节点

登陆到worker主机,直接执行加入worker节点命令即可

kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866

引用