今天我们来做一些比较实际的事情，自动化部署 docker 和 k8s。 之前我写过一篇 kubeadm 安装 k8s 踩坑记， 建议大家先看一下。链接地址：https://testerhome.com/topics/8668
我把每一步的解释放在下面的注释里。
# init docker
# 我们要安装最新的docker-ce版本。 但是docker的坑之一就是变化特别快。 社区曾经有docker 和docker-engine 两个包名，同样的依赖包也有docker-common等。 而这些包都是彼此冲突的。如果机器上装过老版本的docker的话就悲剧了。所以首先要把环境里的痕迹清理干净。把下面的包依次删除
remove docker:
pkg.removed:
- pkgs:
- docker
- docker-ce
- docker-common
- docker-selinux
- docker-engine
- docker-ce-selinux
- container-selinux
# 下面是安装docker-ce 按照官网推荐的centos方式进行安装。
install docker-ce:
cmd.run:
- name: yum install -y yum-utils device-mapper-persistent-data lvm2; yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo; yum -y install docker-ce
- require:
- pkg: remove docker
# 把docker.service 复制到节点上。 这是systemd控制docker的配置文件，我们做了定制例如使用aufs驱动，cgroups driver使用cgroupfs， 配置阿里加速器等等。后面会把这个文件贴出来。
docker_service:
file.managed:
- name: /lib/systemd/system/docker.service
- source: salt://utils/k8s/docker.service
- user: root
- group: root
- mode: 644
- require:
- cmd: install docker-ce
# 配置并启动docker
enable_docker:
cmd.run:
- name: rm -f /etc/docker/daemon.json; systemctl disable docker; systemctl enable docker; systemctl start docker; systemctl restart docker; systemctl status docker
- require:
- file: docker_service
# 要连接公司内部的私有镜像仓库，需要有login后的认证文件。 同样copy到节点上。
docker_login:
cmd.run:
- name: mkdir -p /root/.docker/
file.managed:
- name: /root/.docker/config.json
- source: salt://utils/k8s/config.json
- require:
- cmd: docker_login
下面是 docker.service 的配置：
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target firewalld.service
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd --registry-mirror=https://5f6xdooq.mirror.aliyuncs.com --live-restore=true --exec-opt native.cgroupdriver=cgroupfs --storage-driver=aufs
ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# 引入上面安装docker的文件
include:
- init_docker
# copy kubeadm的源，这里使用阿里源
kubeadm.repo:
file.managed:
- name: /etc/yum.repos.d/kubernetes.repo
- source: salt://utils/k8s/kubernetes.repo
- user: root
- group: root
- mode: 644
# 安装kubadm
install kubeadm:
pkg.installed:
- name: kubeadm
- require:
- file: kubeadm.repo
# 启动前清理iptables，避免一些奇怪的规则堵死集群网络
init_iptalbes:
cmd.run:
- name: iptables -P INPUT ACCEPT; iptables -F; iptables -X; iptables -Z
- require:
- pkg: install kubeadm
# 将kubelet的cgroup-driver从systemd修改为cgroupfs。 kubelet默认使用systemd。但是刚才在docker的配置中我们指定了cgroupfs，这两者必须保持一致。 为什么不使用systemd呢？因为后面安装flannel网络的时候，flannel在systemd下会报错。
set_kubelet:
cmd.run:
- name: sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
- require:
- pkg: install kubeadm
# 引入安装kubeadm的文件
include:
- init_kubeadm
- tools
# 初始化镜像， 由于google被墙，我实现将需要的镜像放到了公司的镜像仓库中。需要在master节点上把这些镜像拉下来，并修改成gcr的名字才可使用
k8s_images_init:
cmd.run:
- name: docker pull registry.4paradigm.com/pause-amd64:3.0; docker tag registry.4paradigm.com/pause-amd64:3.0 gcr.io/google_containers/pause-amd64:3.0; docker pull registry.4paradigm.com/kube-proxy-amd64:v1.8.2; docker tag registry.4paradigm.com/kube-proxy-amd64:v1.8.2 gcr.io/google_containers/kube-proxy-amd64:v1.8.2; docker pull registry.4paradigm.com/nginx-ingress-controller:0.9.0; docker tag registry.4paradigm.com/nginx-ingress-controller:0.9.0 quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0; docker pull registry.4paradigm.com/k8s-dns-sidecar-amd64:1.14.4; docker tag registry.4paradigm.com/k8s-dns-sidecar-amd64:1.14.4 gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.4; docker pull registry.4paradigm.com/kube-apiserver-amd64:v1.8.2; docker tag registry.4paradigm.com/kube-apiserver-amd64:v1.8.2 gcr.io/google_containers/kube-apiserver-amd64:v1.8.2; docker pull registry.4paradigm.com/kube-controller-manager-amd64:v1.8.2; docker tag registry.4paradigm.com/kube-controller-manager-amd64:v1.8.2 gcr.io/google_containers/kube-controller-manager-amd64:v1.8.2; docker pull registry.4paradigm.com/kube-scheduler-amd64:v1.8.2; docker tag registry.4paradigm.com/kube-scheduler-amd64:v1.8.2 gcr.io/google_containers/kube-scheduler-amd64:v1.8.2; docker pull registry.4paradigm.com/defaultbackend:1.4; docker tag registry.4paradigm.com/defaultbackend:1.4 gcr.io/google_containers/defaultbackend:1.4; docker pull registry.4paradigm.com/k8s-dns-kube-dns-amd64:1.14.4; docker tag registry.4paradigm.com/k8s-dns-kube-dns-amd64:1.14.4 gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.4; docker pull registry.4paradigm.com/k8s-dns-dnsmasq-nanny-amd64:1.14.4; docker tag registry.4paradigm.com/k8s-dns-dnsmasq-nanny-amd64:1.14.4 gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.4; docker pull registry.4paradigm.com/etcd-amd64:3.0.17; docker tag registry.4paradigm.com/etcd-amd64:3.0.17 gcr.io/google_containers/etcd-amd64:3.0.17
- require:
- file: docker_login
- cmd: enable_docker
# 初始化集群
setup_master:
cmd.run:
- name: kubeadm reset; kubeadm init --pod-network-cidr=10.244.0.0/16 --kubernetes-version=v1.8.2 --token-ttl 0
- require:
- cmd: k8s_images_init
- pkg: install kubeadm
# 配置kubectl，把需要的kube config 文件放到root用户下，以便root用户可以使用该命令
setup_kubectl:
cmd.run:
- name: mkdir -p /root/.kube; rm -f /root/.kube/config/; cp /etc/kubernetes/admin.conf /root/.kube/config;chown $(id -u):$(id -g) /root/.kube/config;
- require:
- cmd: setup_master
# 下载portmap，这是隧道网络flannel需要用的组件，很多系统上没有这个组件，需要事先安装好
port_map:
file.managed:
- name: /opt/cni/bin/portmap
- source: salt://utils/k8s/portmap
- user: root
- group: root
- mode: 777
# 初始化集群网络，这里使用flannel
install_flannel:
file.managed:
- name: /root/kube-flannel.yml
- source: salt://utils/k8s/kube-flannel.yml
- user: root
- group: root
- mode: 644
- require:
- cmd: setup_master
cmd.run:
- name: kubectl create -f /root/kube-flannel.yml
- require:
- file: install_flannel
- cmd: setup_kubectl
- file: port_map
# 首先创建测试服务专用的namespace
test-service_namespace:
cmd.run:
- name: kubectl create namespace test-service
# 为default namespace设置image secret，k8s只有拥有这个secret才能在镜像仓库上下载镜像
default_image_secret:
file.managed:
- name: /root/image-pull-secret_default.yaml
- source: salt://utils/k8s/image-pull-secret_default.yaml
- user: root
- group: root
- mode: 644
cmd.run:
- name: kubectl create -f /root/image-pull-secret_default.yaml
# 同上，为test-service 这个namespace设置secret
test-service_image_secret:
file.managed:
- name: /root/image-pull-secret_test-service.yaml
- source: salt://utils/k8s/image-pull-secret_test-service.yaml
- user: root
- group: root
- mode: 644
cmd.run:
- name: kubectl create -f /root/image-pull-secret_test-service.yaml
# 安装ingress 服务，这是一个7层路由。
ingress:
file.managed:
- name: /root/ingress.yaml
- source: salt://utils/k8s/ingress.yaml
- user: root
- group: root
- mode: 644
cmd.run:
- name: kubectl create -f /root/ingress.yaml
# 解绑master节点， 默认master节点是不能部署私有的pod的，需要解绑后才可使用
untaint_master:
cmd.run:
- name: export node=`kubectl get nodes|grep Ready|awk '{print $1}'`; kubectl taint nodes $node node-role.kubernetes.io/master-
# 需要运行时通过pillar传递master节点的地址和token
{% set k8s_token = pillar.get("token", "token") %}
{% set master_node = pillar.get("master_node", "token") %}
# 引入安装kubeadm的文件
include:
- init_kubeadm
# 初始化从节点需要的镜像
k8s_images_init:
cmd.run:
- name: docker pull registry.4paradigm.com/pause-amd64:3.0; docker tag registry.4paradigm.com/pause-amd64:3.0 gcr.io/google_containers/pause-amd64:3.0; docker pull registry.4paradigm.com/kube-proxy-amd64:v1.8.2; docker tag registry.4paradigm.com/kube-proxy-amd64:v1.8.2 gcr.io/google_containers/kube-proxy-amd64:v1.8.2
- require:
- file: docker_login
- cmd: enable_docker
# join master节点
join_master:
cmd.run:
- name: kubeadm reset; kubeadm join --token {{k8s_token}} {{master_node}}:6443
- require:
- cmd: k8s_images_init
- pkg: install kubeadm
这就是一个 salt 在初始化 k8s 节点的应用