使用kind搭建kubernetes
<h2 id="使用kind搭建kubernetes">使用kind搭建kubernetes</h2><p></p><div class="toc"><div class="toc-container-header">目录</div><ul><li>使用kind搭建kubernetes<ul><li>kind架构</li><li>创建集群</li><li>将镜像加载到kind的node中</li><li>配置kind集群<ul><li>配置多节点</li><li>多控制面</li><li>指定Kubernetes的版本</li><li>将node的端口映射到主机</li></ul></li><li>ingress部署<ul><li>创建集群</li><li>部署ingress控制器<ul><li>测试ingress</li></ul></li></ul></li><li>总结：</li><li>FAQ：</li></ul></li></ul></div><p></p>
<p>kind是一个非常方便的用于创建kubernetes测试集群的工具，可以使用kind创建的集群来对kubernetes进行测试。</p>
<h3 id="kind架构">kind架构</h3>
<p>kind的架构如下，它将docker容器作为一个kubernetes的"node"，并在该"node"中安装kubernetes组件。</p>
<p><img src="https://img2020.cnblogs.com/blog/1334952/202009/1334952-20200923134115057-2097644433.png"></p>
<p>虚拟机和kind的<strong>一个</strong>"node"之间的网络模型如下，多"node"之间的通信走bridge即可。</p>
<p><img src="https://img2020.cnblogs.com/blog/1334952/202009/1334952-20200924165017436-1143752029.png"></p>
<p>虚拟机到node容器使用了docker的bridge网络(见下面的kind网桥)</p>
<pre><code class="language-shell"># docker network ls
NETWORK ID NAME DRIVER SCOPE
0912809efeab bridge bridge local
3b3ab0cdd358 host host local
198d0909f9cf kind bridge local
b0e304ef061c none null local
</code></pre>
<p>在kind的"node"内部则默认使用了ptp模型，该模型的通信机制比较简单，容器和"node"直接通过一对veth进行通信。</p>
<pre><code class="language-json"># cat /etc/cni/net.d/10-kindnet.conflist
{
"cniVersion": "0.3.1",
"name": "kindnet",
"plugins": [
{
"type": "ptp",
"ipMasq": false,
"ipam": {
"type": "host-local",
"dataDir": "/run/cni-ipam-state",
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"ranges": [
[
{
"subnet": "10.244.0.0/24"
}
]
]
}
,
"mtu": 1500
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
</code></pre>
<p>"node"内的路由如下：10.244.0.2~0.4的地址就是"node"中的容器的eth0网卡上的地址，通过位于"node"上的veth进行通信。但"node"中的容器并不都是使用ptp网络模型，如kubernetes的apiserver，scheduler和controller-manager就是用了host模式。</p>
<pre><code class="language-shell"># ip route
default via 172.18.0.1 dev eth0
10.244.0.2 dev vethc85f5947 scope host
10.244.0.3 dev vetheb946a41 scope host
10.244.0.4 dev veth46c994ce scope host
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.2
</code></pre>
<h3 id="创建集群">创建集群</h3>
<ul>
<li>
<p>安装kubectl</p>
</li>
<li>
<p>安装kind命令：</p>
<pre><code class="language-shell">curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.9.0/kind-linux-amd64
chmod +x ./kind
mv ./kind /${some-dir-in-your-PATH}/kind
</code></pre>
</li>
<li>
<p>创建集群，默认的集群名称为<code>kind</code>，可以使用参数<code>--name</code>指定创建的集群的名称，多集群情况下比较有用</p>
<pre><code class="language-shell">kind create cluster
</code></pre>
<p>也指定node镜像来创建集群，便于离线安装</p>
<pre><code class="language-shell">kind create cluster --image kindest/node:latest
</code></pre>
</li>
<li>
<p>与集群交互：</p>
<ol>
<li>
<p>获取集群名称，可以看到下面有两个集群</p>
<pre><code class="language-shell"># kind get clusters
kind
kind-2
</code></pre>
</li>
<li>
<p>切换集群。可以使用如下命令分别切换到集群<code>kind</code>和<code>kind-2</code></p>
<pre><code class="language-shell"># kubectl cluster-info --context kind-kind
# kubectl cluster-info --context kind-kind-2
</code></pre>
</li>
</ol>
</li>
<li>
<p>删除集群，如使用如下命令可以删除集群<code>kind-2</code></p>
<pre><code>kind delete cluster --name kind-2
</code></pre>
</li>
</ul>
<h3 id="将镜像加载到kind的node中">将镜像加载到kind的node中</h3>
<p>kind创建的kubernetes会使用它的node上的镜像，因此需要将将镜像加载到node中才能被kubernetes使用(当然在node中也是可以直接拉取公网镜像的)，在无法拉取公网镜像的时候可以手动将镜像load到node上使用。例如，使用如下方式可以将nginx镜像加载到名为kind的集群中：</p>
<pre><code class="language-shell"># kind load docker-image nginx --name kind
Image: "nginx" with ID "sha256:7e4d58f0e5f3b60077e9a5d96b4be1b974b5a484f54f9393000a99f3b6816e3d" not yet present on node "kind-control-plane", loading...
</code></pre>
<h3 id="配置kind集群">配置kind集群</h3>
<p>可以在kind创建集群的时候使用配置文件进行自定义配置。例如可以使用--config指定配置文件来创建集群：</p>
<pre><code class="language-shell">kind create cluster --config kind-example-config.yaml
</code></pre>
<h4 id="配置多节点">配置多节点</h4>
<p>上面部署的kubernetes中只有一个node，可以使用配置文件部署多个节点。下面使用官方提供的默认配置文件<code>kind-config.yaml</code>来创建集群，该集群含3个work节点：</p>
<pre><code class="language-yaml"># this config file contains all config fields with comments
# NOTE: this is not a particularly useful config file
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
# patch the generated kubeadm config with some extra settings
kubeadmConfigPatches:
- |
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
evictionHard:
nodefs.available: "0%"
# patch it further using a JSON 6902 patch
kubeadmConfigPatchesJSON6902:
- group: kubeadm.k8s.io
version: v1beta2
kind: ClusterConfiguration
patch: |
- op: add
path: /apiServer/certSANs/-
value: my-hostname
# 1 control plane node and 3 workers
nodes:
# the control plane node config
- role: control-plane
# the three workers
- role: worker
- role: worker
- role: worker
</code></pre>
<p>创建上述集群：</p>
<pre><code class="language-shell"># kind create cluster --name multi-node --config=kind-config.yaml
</code></pre>
<p>切换到该集群：</p>
<pre><code class="language-shell"># kubectl cluster-info --context kind-multi-node
</code></pre>
<p>可以看到该集群下有1个控制面node，以及3个work node：</p>
<pre><code class="language-shell"># kubectl get node
NAME STATUS ROLES AGE VERSION
multi-node-control-plane Ready master 7m57s v1.19.1
multi-node-worker Ready <none> 7m21s v1.19.1
multi-node-worker2 Ready <none> 7m21s v1.19.1
multi-node-worker3 Ready <none> 7m21s v1.19.1
</code></pre>
<h4 id="多控制面">多控制面</h4>
<p>一般一个生产使用的kubernetes都会使用多个控制面来保证高可用，使用kind config可以方便地创建多控制面的kubernetes集群。使用如下命令创建一个3控制面，3 work节点的集群：</p>
<pre><code class="language-yaml"># this config file contains all config fields with comments
# NOTE: this is not a particularly useful config file
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
# patch the generated kubeadm config with some extra settings
kubeadmConfigPatches:
- |
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
evictionHard:
nodefs.available: "0%"
# patch it further using a JSON 6902 patch
kubeadmConfigPatchesJSON6902:
- group: kubeadm.k8s.io
version: v1beta2
kind: ClusterConfiguration
patch: |
- op: add
path: /apiServer/certSANs/-
value: my-hostname
# 1 control plane node and 3 workers
nodes:
# the control plane node config
- role: control-plane
- role: control-plane
- role: control-plane
# the three workers
- role: worker
- role: worker
- role: worker
</code></pre>
<p>此时可以看到有3个控制面：</p>
<pre><code class="language-shell"># kubectl get node
NAME STATUS ROLES AGE VERSION
kind-control-plane Ready master 15m v1.19.1
kind-control-plane2 Ready master 14m v1.19.1
kind-control-plane3 Ready master 13m v1.19.1
kind-worker Ready <none> 12m v1.19.1
kind-worker2 Ready <none> 12m v1.19.1
kind-worker3 Ready <none> 12m v1.19.1
</code></pre>
<h4 id="指定kubernetes的版本">指定Kubernetes的版本</h4>
<p>可以通过指定node的镜像版本来修改kubernetes的版本。可以在官方release页面中中查找需要镜像tag，推荐tag带上sha，如</p>
<p><code>kindest/node:v1.19.1@sha256:98cf5288864662e37115e362b23e4369c8c4a408f99cbc06e58ac30ddc721600</code></p>
<pre><code class="language-yaml">kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
image: kindest/node:v1.16.4@sha256:b91a2c2317a000f3a783489dfb755064177dbc3a0b2f4147d50f04825d016f55
- role: worker
image: kindest/node:v1.16.4@sha256:b91a2c2317a000f3a783489dfb755064177dbc3a0b2f4147d50f04825d016f55
</code></pre>
<h4 id="将node的端口映射到主机">将node的端口映射到主机</h4>
<p>可以通过如下方式将node的端口映射到主机，将容器的80端口映射到host的80端口：</p>
<pre><code class="language-yaml">kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 80
hostPort: 80
listenAddress: "0.0.0.0" # Optional, defaults to "0.0.0.0"
protocol: udp # Optional, defaults to tcp
</code></pre>
<p>kind对<code>cluster</code>的更新(如启用IPv6，配置nodeport等)有一个弊端，就是只能通过重新创建集群来"更新"配置。目前官方不支持对控制面的更新操作，可以参见该issue。更多配置参见官方文档。</p>
<h3 id="ingress部署">ingress部署</h3>
<p>可以通过KIND的extraPortMapping配置选项来将流量从主机转发到node的ingress控制器上。</p>
<p>可以通过kubeadm的<code>InitConfiguration</code>来设置自定义的<code>node-labels</code>，用于ingress控制器的<code>nodeSelector</code>。</p>
<h4 id="创建集群-1">创建集群</h4>
<p>使用<code>extraPortMappings</code> 和<code>node-labels</code>创建一个集群。</p>
<pre><code class="language-yaml">cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "ingress-ready=true"
extraPortMappings:
- containerPort: 80
hostPort: 80
protocol: TCP
- containerPort: 443
hostPort: 443
protocol: TCP
EOF
</code></pre>
<h4 id="部署ingress控制器">部署ingress控制器</h4>
<p>kind支持的ingress控制器如下：</p>
<ul>
<li>Ambassador</li>
<li>Contour</li>
<li>Ingress NGINX</li>
</ul>
<p>下面部署NGINX ingress。</p>
<pre><code class="language-shell">kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/kind/deploy.yaml
</code></pre>
<p>在部署ingree的过程中可能会遇到无法找到secret <code>ingress-nginx-admission</code>的问题，出现该问题的原因可能是因为如下两个job无法正常启动造成的，参见该issue。如果是因为无法拉取外网镜像，可以先将deploy.yaml文件下载到本地，将镜像手动加载到本地主机上，然后使用上面提到的<code>kind load docker-image</code>命令将镜像加载到node上即可。</p>
<p><img src="https://img2020.cnblogs.com/blog/1334952/202009/1334952-20200925113948778-455183541.png"></p>
<h5 id="测试ingress">测试ingress</h5>
<p>创建如下资源：<code>kubectl apply -f usage.yaml</code></p>
<pre><code class="language-yaml">kind: Pod
apiVersion: v1
metadata:
name: foo-app
labels:
app: foo
spec:
containers:
- name: foo-app
image: hashicorp/http-echo:0.2.3
args:
- "-text=foo"
---
kind: Service
apiVersion: v1
metadata:
name: foo-service
spec:
selector:
app: foo
ports:
# Default port used by the image
- port: 5678
---
kind: Pod
apiVersion: v1
metadata:
name: bar-app
labels:
app: bar
spec:
containers:
- name: bar-app
image: hashicorp/http-echo:0.2.3
args:
- "-text=bar"
---
kind: Service
apiVersion: v1
metadata:
name: bar-service
spec:
selector:
app: bar
ports:
# Default port used by the image
- port: 5678
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: example-ingress
spec:
rules:
- http:
paths:
- path: /foo
backend:
serviceName: foo-service
servicePort: 5678
- path: /bar
backend:
serviceName: bar-service
servicePort: 5678
---
</code></pre>
<p>在远端curl该主机所在上的foo和bar服务，可以看到网络是通的，此时走的是ingress通过kind配置<code>extraPortMappings</code>暴露的nodeport 80端口。</p>
<pre><code class="language-bash">C:\Users\liuch>curl 192.168.100.11/foo
foo
C:\Users\liuch>curl 192.168.100.11/bar
bar
</code></pre>
<h3 id="总结">总结：</h3>
<p>kind是一个非常方便的kubernetes部署工具，可以快速地部署多个kubernetes集群。但也有一些实现上的瑕疵，比如，kind不支持对集群的升级，手动加载镜像的过程也比较麻烦，但总体使用上来看，瑕不掩瑜。</p>
<h3 id="faq">FAQ：</h3>
<ul>
<li>
<p>在切换集群时出现<code>The connection to the server localhost:8080 was refused - did you specify the right host or port?</code>，且使用如<code>kubectl config use-context kind-kind</code>这样的命令也无法成功切换集群。</p>
<p>可以在/root/.kube/config文件中查看支持的context名称(如下面使用的context为<code>kind-kind</code>)，然后使用<code>kubectl config use-context kind-kind</code>即可:</p>
<pre><code class="language-yaml">apiVersion: v1
clusters:
- cluster:
certificate-authority-data: ...
server: https://127.0.0.1:39923
name: kind-kind
contexts:
- context:
cluster: kind-kind
user: kind-kind
name: kind-kind
current-context: kind-kind
kind: Config
preferences: {}
users:
- name: kind-kind
user:
client-certificate-data: ...
</code></pre>
</li>
<li>
<p>kind创建的apiservice默认地址是<code>127.0.0.1</code>，无法远程连接。可以使用如下方式，修改apiservice的地址和端口</p>
<pre><code class="language-yaml">kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
# WARNING: It is _strongly_ recommended that you keep this the default
# (127.0.0.1) for security reasons. However it is possible to change this.
apiServerAddress: "192.168.118.154"
# By default the API server listens on a random open port.
# You may choose a specific port but probably don't need to in most cases.
# Using a random port makes it easier to spin up multiple clusters.
apiServerPort: 6443
</code></pre>
</li>
<li>
<p>如果kind无法连通，可以查看是否有其他docker network影响</p>
</li>
</ul>
</div>
<div id="MySignature" role="contentinfo">
<p>本文来自博客园，作者：charlieroro，转载请注明原文链接：https://www.cnblogs.com/charlieroro/p/13711589.html</p><br><br>
来源：https://www.cnblogs.com/charlieroro/p/13711589.html
頁:
[1]