雪碧可乐 發表於 2019-11-7 12:48:00

CentOS 7 ETCD集群配置大全

<p></p><div class="toc"><div class="toc-container-header">目录</div><ul><li>前言</li><li>环境准备</li><li>安装</li><li>静态集群<ul><li>配置<ul><li>node01 配置文件</li><li>node02 配置文件</li><li>node03 配置文件</li></ul></li><li>启动测试<ul><li>查看集群状态</li></ul></li></ul></li><li>生成TLS证书<ul><li>etcd证书创建<ul><li>安装cfssl工具集</li><li>生成证书</li><li>分发证书到各节点上</li></ul></li></ul></li><li>静态TLS集群<ul><li>etcd 配置<ul><li>node01 配置文件</li><li>node02 配置文件</li><li>node03 配置文件</li></ul></li><li>启动测试<ul><li>检查TLS集群状态</li></ul></li></ul></li><li>ETCD 动态集群基于DNS的SRV解析自动发现<ul><li>添加SRV解析<ul><li>方法一: 使用<code>bind</code>配置SRV解析</li><li>方法二: 使用<code>dnsmasq</code>配置SRV解析</li><li>验证SRV解析是否正常</li></ul></li><li>配置ETCD<ul><li>node01 配置文件</li><li>node02 配置文件</li><li>node03 配置文件</li></ul></li><li>启动并测试</li></ul></li><li>ETCD TLS动态集群基于DNS的SRV解析自动发现<ul><li>添加SRV解析<ul><li>方法一: 使用<code>bind</code>配置SRV解析</li><li>方法二: 使用<code>dnsmasq</code>配置SRV解析</li><li>验证SRV解析是否正常</li></ul></li><li>ETCD 配置<ul><li>node01 配置文件</li><li>node02 配置文件</li><li>node03 配置文件</li></ul></li><li>启动测试</li></ul></li><li>报错解决<ul><li>1. 证书报错 bad certificate<ul><li>解决</li></ul></li><li>2. <code>DNS</code> 的 <code>SRV</code> 解析报错 cannot find local etcd member "etcd1" in SRV records<ul><li>http不带证书解析如下<ul><li>bind 的解析</li><li>dnsmasq 的解析</li></ul></li><li>https带证书解析如下<ul><li>bind 的解析</li><li>dnsmasq 的解析</li></ul></li></ul></li></ul></li></ul></div><p></p>
<h1 id="前言">前言</h1>
<blockquote>
<p>Etcd 是 CoreOS 基于 Raft 开发的分布式 key-value 存储，可用于服务发现、共享配置以及一致性保障（如数据库选主、分布式锁等）</p>
</blockquote>
<p>本次环境，是用于k8s集群，由于在二进制部署 k8s 中，由于 Etcd 集群导致各种各样的问题，特意抽出时间来研究 Etcd 集群。</p>
<p>Etcd 集群配置分为三种:</p>
<ol>
<li>静态发现</li>
<li>Etcd 动态发现</li>
<li>DNS 动态发现 通过DNS的SRV解析动态发现集群</li>
</ol>
<p><strong>本次主要基于 静态发现 和 DNS动态发现 两种，并结合自签的TLS证书来创建集群。</strong></p>
<h1 id="环境准备">环境准备</h1>
<p>此环境实际用于 k8s 中的ETCD集群使用，用于本次文档</p>
<table>
<thead>
<tr>
<th>主机名</th>
<th>角色</th>
<th>IP</th>
<th>系统版本</th>
<th>内核版本</th>
</tr>
</thead>
<tbody>
<tr>
<td>node01.k8s.com</td>
<td>node01</td>
<td>192.168.1.91</td>
<td>CentOS 7.7</td>
<td>5.1.4-1.el7.elrepo.x86_64</td>
</tr>
<tr>
<td>node02.k8s.com</td>
<td>node02</td>
<td>192.168.1.92</td>
<td>CentOS 7.7</td>
<td>5.1.4-1.el7.elrepo.x86_64</td>
</tr>
<tr>
<td>node03.k8s.com</td>
<td>node03</td>
<td>192.168.1.93</td>
<td>CentOS 7.7</td>
<td>5.1.4-1.el7.elrepo.x86_64</td>
</tr>
</tbody>
</table>
<h1 id="安装">安装</h1>
<p>在三台机器上均执行</p>
<pre><code># yum install etcd -y
# rpm -qa etcd
etcd-3.3.11-2.el7.centos.x86_64
</code></pre>
<p>创建Etcd所需目录，在三台机器上均执行</p>
<pre><code>mkdir /data/k8s/etcd/{data,wal} -p
mkdir -p /etc/kubernetes/cert
chown -R etcd.etcd /data/k8s/etcd
</code></pre>
<h1 id="静态集群">静态集群</h1>
<h2 id="配置">配置</h2>
<h3 id="node01-配置文件">node01 配置文件</h3>
<pre><code>ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://192.168.1.91:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.1.91:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd1"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.91:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.91:2379"

ETCD_INITIAL_CLUSTER="etcd1=http://192.168.1.91:2380,etcd2=http://192.168.1.92:2380,etcd3=http://192.168.1.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
</code></pre>
<h3 id="node02-配置文件">node02 配置文件</h3>
<pre><code>ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://192.168.1.92:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.1.92:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd2"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.92:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.92:2379"

ETCD_INITIAL_CLUSTER="etcd1=http://192.168.1.91:2380,etcd2=http://192.168.1.92:2380,etcd3=http://192.168.1.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
</code></pre>
<h3 id="node03-配置文件">node03 配置文件</h3>
<pre><code>ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://192.168.1.93:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.1.93:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd3"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.93:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.93:2379"

ETCD_INITIAL_CLUSTER="etcd1=http://192.168.1.91:2380,etcd2=http://192.168.1.92:2380,etcd3=http://192.168.1.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
</code></pre>
<h2 id="启动测试">启动测试</h2>
<pre><code># systemctl start etcd
# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-11-07 09:28:54 CST; 5s ago
Main PID: 1546 (etcd)
    Tasks: 8
   Memory: 41.3M
   CGroup: /system.slice/etcd.service
         └─1546 /usr/bin/etcd --name=etcd1 --data-dir=/data/k8s/etcd/data --listen-client-urls=http://192.168.1.91:2379

Nov 07 09:28:54 node01.k8s.com etcd: 3b8b38de05e2c497 received a MsgVote message with higher term from 9c64fba479c5e94
Nov 07 09:28:54 node01.k8s.com etcd: 3b8b38de05e2c497 became follower at term 2
Nov 07 09:28:54 node01.k8s.com etcd: 3b8b38de05e2c497 cast MsgVote for 9c64fba479c5e94 at term 2
Nov 07 09:28:54 node01.k8s.com etcd: raft.node: 3b8b38de05e2c497 elected leader 9c64fba479c5e94 at term 2
Nov 07 09:28:54 node01.k8s.com etcd: published {Name:etcd1 ClientURLs:} to cluster 19456f0bfd57284e
Nov 07 09:28:54 node01.k8s.com etcd: ready to serve client requests
Nov 07 09:28:54 node01.k8s.com etcd: serving insecure client requests on 192.168.1.91:2379, this is strongly discouraged!
Nov 07 09:28:54 node01.k8s.com systemd: Started Etcd Server.
Nov 07 09:28:54 node01.k8s.com etcd: set the initial cluster version to 3.3
Nov 07 09:28:54 node01.k8s.com etcd: enabled capabilities for version 3.3

</code></pre>
<p>查看 /var/log/message 日志中，会有日下体现：</p>
<pre><code>Nov7 09:28:53 node02 etcd: added member 9c64fba479c5e94 to cluster 19456f0bfd57284e
Nov7 09:28:53 node02 etcd: added member 3b8b38de05e2c497 to cluster 19456f0bfd57284e
Nov7 09:28:53 node02 etcd: added member 76ea8679db7365b3 to cluster 19456f0bfd57284e
</code></pre>
<h3 id="查看集群状态">查看集群状态</h3>
<pre><code># ETCDCTL_API=3 etcdctl --endpoints=http://192.168.1.91:2379,http://192.168.1.92:2379,http://192.168.1.93:2379 endpoint health
http://192.168.1.92:2379 is healthy: successfully committed proposal: took = 1.103545ms
http://192.168.1.93:2379 is healthy: successfully committed proposal: took = 2.122478ms
http://192.168.1.91:2379 is healthy: successfully committed proposal: took = 2.690215ms
</code></pre>
<pre><code># etcdctl --endpoints=http://192.168.1.91:2379,http://192.168.1.92:2379,http://192.168.1.93:2379 cluster-health
member 9c64fba479c5e94 is healthy: got healthy result from http://192.168.1.92:2379
member 3b8b38de05e2c497 is healthy: got healthy result from http://192.168.1.91:2379
member 76ea8679db7365b3 is healthy: got healthy result from http://192.168.1.93:2379
cluster is healthy
</code></pre>
<h1 id="生成tls证书">生成TLS证书</h1>
<p><strong>使用自签证书</strong></p>
<blockquote>
<p>CA(Certificate Authority)是自签名的根证书，用来签名后续创建的其他证书。本文章使用CloudFlare的PKI工具cfssl创建所有证书。</p>
</blockquote>
<h2 id="etcd证书创建">etcd证书创建</h2>
<p><strong>整个证书的创建过程均在 <code>node01</code> 上操作;</strong></p>
<h3 id="安装cfssl工具集">安装cfssl工具集</h3>
<pre><code>mkdir -p /opt/k8s/cert &amp;&amp; cd /opt/k8s
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
mv cfssl_linux-amd64 /opt/k8s/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
mv cfssljson_linux-amd64 /opt/k8s/bin/cfssljson
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /opt/k8s/bin/cfssl-certinfo
chmod +x /opt/k8s/bin/*
echo 'export PATH=/opt/k8s/bin:$PATH' &gt;&gt; ~/.bash_profile
source ~/.bash_profile
</code></pre>
<h3 id="生成证书">生成证书</h3>
<p><strong>创建根证书 (CA)</strong></p>
<blockquote>
<p>CA证书是集群所有节点共享的，只需要创建一个CA证书，后续创建的所有证书都是由它签名</p>
</blockquote>
<p><strong>创建配置文件</strong></p>
<blockquote>
<p>CA配置文件用于配置根证书的使用场景(profile)和具体参数</p>
</blockquote>
<blockquote>
<p>(usage、过期时间、服务端认证、客户端认证、加密等)</p>
</blockquote>
<pre><code>cd /opt/k8s/work
cat &gt; ca-config.json &lt;&lt;EOF
{
    "signing": {
      "default": {
            "expiry": "876000h"
      },
      "profiles": {
            "server": {
                "expiry": "876000h",
                "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
                ]
            },
            "client": {
                "expiry": "876000h",
                "usages": [
                  "signing",
                  "key encipherment",
                  "client auth"
                ]
            },
            "peer": {
                "expiry": "876000h",
                "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
                ]
            }
      }
    }
}
EOF

######################
signing 表示该证书可用于签名其它证书，生成的ca.pem证书找中CA=TRUE
server auth 表示client可以用该证书对server提供的证书进行验证
client auth 表示server可以用该证书对client提供的证书进行验证
</code></pre>
<p><strong>创建证书签名请求文件</strong></p>
<pre><code>cd /opt/k8s/work
cat &gt; ca-csr.json &lt;&lt;EOF
{
    "CN": "etcd",
    "key": {
      "algo": "rsa",
      "size": 2048
    }
}
EOF


#######################
CN CommonName,kube-apiserver从证书中提取该字段作为请求的用户名(User Name)，浏览器使用该字段验证网站是否合法
O Organization,kube-apiserver 从证书中提取该字段作为请求用户和所属组(Group)
kube-apiserver将提取的User、Group作为RBAC授权的用户和标识
</code></pre>
<p><strong>生成CA证书和私钥</strong></p>
<pre><code>cd /opt/k8s/work
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ls ca*
</code></pre>
<p><strong>生成客户端证书</strong></p>
<pre><code>cat &gt;client.json&lt;&lt;EOF
{
    "CN": "client",
    "key": {
      "algo": "ecdsa",
      "size": 256
    }
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client.json| cfssljson -bare client -

</code></pre>
<p><strong>创建etcd证书和私钥</strong></p>
<pre><code>cd /opt/k8s/work
cat &gt; etcd-csr.json &lt;&lt;EOF
{
"CN": "etcd",
"hosts": [
    "127.0.0.1",
    "192.168.1.91",
    "192.168.1.92",
    "192.168.1.93",
    "k8s.com",
    "etcd1.k8s.com",
    "etcd2.k8s.com",
    "etcd3.k8s.com"
],
"key": {
    "algo": "rsa",
    "size": 2048
},
"names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
]
}
EOF

#host字段指定授权使用该证书的etcd节点IP或域名列表，需要将etcd集群的3个节点都添加其中
</code></pre>
<p><strong>生成证书和私钥</strong></p>
<pre><code>cd /opt/k8s/work

cfssl gencert -ca=/opt/k8s/work/ca.pem \
    -ca-key=/opt/k8s/work/ca-key.pem \
    -config=/opt/k8s/work/ca-config.json \
    -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
   
ls etcd*pem -l
-rw------- 1 root root 1675 Nov7 09:52 etcd-key.pem
-rw-r--r-- 1 root root 1444 Nov7 09:52 etcd.pem
</code></pre>
<p><strong>etcd 使用的TLS证书创建完成</strong></p>
<h3 id="分发证书到各节点上">分发证书到各节点上</h3>
<p><strong>要做所有节点上创建对应的目录</strong></p>
<pre><code>mkdir /data/k8s/etcd/{data,wal} -p
mkdir -p /etc/kubernetes/cert
chown -R etcd.etcd /data/k8s/etcd
</code></pre>
<p><strong>分发证书</strong></p>
<pre><code>cd /opt/k8s/work
scp ca*.pem ca-config.json 192.168.1.91:/etc/kubernetes/cert
scp ca*.pem ca-config.json 192.168.1.92:/etc/kubernetes/cert
scp ca*.pem ca-config.json 192.168.1.93:/etc/kubernetes/cert
scp etcd*pem 192.168.1.91:/etc/etcd/cert/
scp etcd*pem 192.168.1.92:/etc/etcd/cert/
scp etcd*pem 192.168.1.93:/etc/etcd/cert/
</code></pre>
<p><strong>在所有节点上执行:</strong></p>
<pre><code>chown -R etcd.etcd /etc/etcd/cert
</code></pre>
<h1 id="静态tls集群">静态TLS集群</h1>
<h2 id="etcd-配置">etcd 配置</h2>
<h3 id="node01-配置文件-1">node01 配置文件</h3>
<pre><code>ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="https://192.168.1.91:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.91:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd1"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.91:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.91:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.91:2380,etcd2=https://192.168.1.92:2380,etcd3=https://192.168.1.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_PEER_AUTO_TLS="true"
</code></pre>
<h3 id="node02-配置文件-1">node02 配置文件</h3>
<pre><code>ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="https://192.168.1.92:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.92:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd2"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.92:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.92:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.91:2380,etcd2=https://192.168.1.92:2380,etcd3=https://192.168.1.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_PEER_AUTO_TLS="true"
</code></pre>
<h3 id="node03-配置文件-1">node03 配置文件</h3>
<pre><code>ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="https://192.168.1.93:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.93:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd3"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.93:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.93:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.91:2380,etcd2=https://192.168.1.92:2380,etcd3=https://192.168.1.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_PEER_AUTO_TLS="true"
</code></pre>
<h2 id="启动测试-1">启动测试</h2>
<pre><code># systemctl start etcd
# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-11-07 10:15:58 CST; 5s ago
Main PID: 2078 (etcd)
    Tasks: 8
   Memory: 28.9M
   CGroup: /system.slice/etcd.service
         └─2078 /usr/bin/etcd --name=etcd1 --data-dir=/data/k8s/etcd/data --listen-client-urls=https://192.168.1.91:2379

Nov 07 10:15:58 node01.k8s.com etcd: 2a40d8ba966d12fe received a MsgVote message with higher term from af05139f75a68867
Nov 07 10:15:58 node01.k8s.com etcd: 2a40d8ba966d12fe became follower at term 2
Nov 07 10:15:58 node01.k8s.com etcd: 2a40d8ba966d12fe cast MsgVote for af05139f75a68867 at term 2
Nov 07 10:15:58 node01.k8s.com etcd: raft.node: 2a40d8ba966d12fe elected leader af05139f75a68867 at term 2
Nov 07 10:15:58 node01.k8s.com etcd: published {Name:etcd1 ClientURLs:} to cluster f3e9c54e1aafb3c1
Nov 07 10:15:58 node01.k8s.com etcd: ready to serve client requests
Nov 07 10:15:58 node01.k8s.com etcd: serving client requests on 192.168.1.91:2379
Nov 07 10:15:58 node01.k8s.com systemd: Started Etcd Server.
Nov 07 10:15:58 node01.k8s.com etcd: set the initial cluster version to 3.3
Nov 07 10:15:58 node01.k8s.com etcd: enabled capabilities for version 3.3
</code></pre>
<p>查看 /var/log/message 日志中，会有日下体现：</p>
<pre><code>Nov7 10:15:57 node01 etcd: added member 2a40d8ba966d12fe to cluster f3e9c54e1aafb3c1
Nov7 10:15:57 node01 etcd: added member af05139f75a68867 to cluster f3e9c54e1aafb3c1
Nov7 10:15:57 node01 etcd: added member c3bab7c20fba3f60 to cluster f3e9c54e1aafb3c1
</code></pre>
<h3 id="检查tls集群状态">检查TLS集群状态</h3>
<pre><code>ETCDCTL_API=3 etcdctl \
    --endpoints=https://etcd1.k8s.com:2379,https://etcd2.k8s.com:2379,https://etcd3.k8s.com:2379 \
    --cacert=/etc/kubernetes/cert/ca.pem \
    --cert=/etc/etcd/cert/etcd.pem \
    --key=/etc/etcd/cert/etcd-key.pem endpoint health

# 输出
https://192.168.1.92:2379 is healthy: successfully committed proposal: took = 1.317022ms
https://192.168.1.91:2379 is healthy: successfully committed proposal: took = 1.59958ms
https://192.168.1.93:2379 is healthy: successfully committed proposal: took = 1.453049ms
</code></pre>
<pre><code>etcdctl \
--endpoint=https://etcd1.k8s.com:2379 \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/etcd/cert/etcd.pem \
--key-file=/etc/etcd/cert/etcd-key.pem cluster-health

# 输出
member 40a8f19a5db99534 is healthy: got healthy result from https://etcd2.k8s.com:2379
member 9888555207dbf0e0 is healthy: got healthy result from https://etcd3.k8s.com:2379
member a0d541999e9eb3b3 is healthy: got healthy result from https://etcd1.k8s.com:2379
cluster is healthy
</code></pre>
<h1 id="etcd-动态集群基于dns的srv解析自动发现">ETCD 动态集群基于DNS的SRV解析自动发现</h1>
<blockquote>
<p>需要局域网内部有DNS服务器</p>
</blockquote>
<h2 id="添加srv解析">添加SRV解析</h2>
<blockquote>
<p>目前常用的内部DNS服务有两种，<code>bind</code>、<code>dnsmasq</code></p>
</blockquote>
<p><strong>在下面都会列出具体的配置，但只需要配置其中之一即可；</strong></p>
<h3 id="方法一-使用bind配置srv解析">方法一: 使用<code>bind</code>配置SRV解析</h3>
<p>如果内部没有<code>bind</code>服务，可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11806962.html</p>
<p>使用域名为 : <code>k8s.com</code>，在bind的zone文件中添加如下解析:</p>
<pre><code>etcd1   IN      A       192.168.1.91
etcd2   IN      A       192.168.1.92
etcd3   IN      A       192.168.1.93
_etcd-server._tcp.k8s.com.IN      SRV   10    10       2380   etcd1
_etcd-server._tcp.k8s.com.IN      SRV   10    10       2380   etcd2
_etcd-server._tcp.k8s.com.IN      SRV   10    10       2380   etcd3
_etcd-client._tcp.k8s.com.IN      SRV   10    10       2379   etcd1
_etcd-client._tcp.k8s.com.IN      SRV   10    10       2379   etcd2
_etcd-client._tcp.k8s.com.IN      SRV   10    10       2379   etcd3
</code></pre>
<p>修改之后重新加载配置文件:</p>
<pre><code># named-checkzone k8s.com k8s.com.zone
zone k8s.com/IN: loaded serial 0
OK
# rndc reload
server reload successful
</code></pre>
<h3 id="方法二-使用dnsmasq配置srv解析">方法二: 使用<code>dnsmasq</code>配置SRV解析</h3>
<p>如果内部没有<code>dnsmasq</code>服务，可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11809066.html</p>
<p>使用域名为 : <code>k8s.com</code>，具体修改如下:</p>
<p>在<code>/etc/dnsmasq_hosts</code>新增下面内容</p>
<pre><code>192.168.1.91 etcd1 etcd1.k8s.com
192.168.1.92 etcd2 etcd2.k8s.com
192.168.1.93 etcd3 etcd3.k8s.com
</code></pre>
<p>在 <code>/etc/dnsmasq.conf</code> 文件中增加下面SRV解析内容</p>
<pre><code>srv-host=_etcd-server._tcp.k8s.com,etcd1.k8s.com,2380,0,100
srv-host=_etcd-server._tcp.k8s.com,etcd2.k8s.com,2380,0,100
srv-host=_etcd-server._tcp.k8s.com,etcd3.k8s.com,2380,0,100
srv-host=_etcd-client._tcp.k8s.com,etcd1.k8s.com,2379,0,100
srv-host=_etcd-client._tcp.k8s.com,etcd2.k8s.com,2379,0,100
srv-host=_etcd-client._tcp.k8s.com,etcd3.k8s.com,2379,0,100
</code></pre>
<p>修改之后重启服务 <code>systemctl restart dnsmasq</code></p>
<h3 id="验证srv解析是否正常">验证SRV解析是否正常</h3>
<p><strong>查询SRV记录</strong></p>
<pre><code># dig @192.168.1.122 +noall +answer SRV _etcd-server._tcp.k8s.com
_etcd-server._tcp.k8s.com. 3600        IN        SRV        2380 0 100 etcd2.k8s.com.
_etcd-server._tcp.k8s.com. 3600        IN        SRV        2380 0 100 etcd1.k8s.com.
_etcd-server._tcp.k8s.com. 3600        IN        SRV        2380 0 100 etcd3.k8s.com.
</code></pre>
<p><strong>查询域名解析结果</strong></p>
<pre><code># dig @192.168.1.122 +noall +answer etcd1.k8s.com etcd2.k8s.com etcd3.k8s.com
etcd1.k8s.com.                86400        IN        A        192.168.1.91
etcd2.k8s.com.                86400        IN        A        192.168.1.92
etcd3.k8s.com.                86400        IN        A        192.168.1.93
</code></pre>
<p>如上述显示，则表示SRV解析正常</p>
<h2 id="配置etcd">配置ETCD</h2>
<h3 id="node01-配置文件-2">node01 配置文件</h3>
<pre><code>ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://192.168.1.91:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.1.91:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd1"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd1.k8s.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://etcd1.k8s.com:2379"
ETCD_DISCOVERY_SRV="k8s.com"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
</code></pre>
<h3 id="node02-配置文件-2">node02 配置文件</h3>
<pre><code>ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://192.168.1.92:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.1.92:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd2"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd2.k8s.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://etcd2.k8s.com:2379"
ETCD_DISCOVERY_SRV="k8s.com"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
</code></pre>
<h3 id="node03-配置文件-2">node03 配置文件</h3>
<pre><code>ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://192.168.1.93:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.1.93:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd3"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd3.k8s.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://etcd3.k8s.com:2379"
ETCD_DISCOVERY_SRV="k8s.com"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
</code></pre>
<h2 id="启动并测试">启动并测试</h2>
<p>启动</p>
<pre><code># systemctl start etcd
# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-11-07 11:25:29 CST; 4s ago
Main PID: 14203 (etcd)
    Tasks: 8
   Memory: 16.9M
   CGroup: /system.slice/etcd.service
         └─14203 /usr/bin/etcd --name=etcd1 --data-dir=/data/k8s/etcd/data --listen-client-urls=http://192.168.1.91:2379

Nov 07 11:25:29 node01.k8s.com etcd: d79e9ae86b2a1de1 has received 2 MsgVoteResp votes and 0 vote rejections
Nov 07 11:25:29 node01.k8s.com etcd: d79e9ae86b2a1de1 became leader at term 2
Nov 07 11:25:29 node01.k8s.com etcd: raft.node: d79e9ae86b2a1de1 elected leader d79e9ae86b2a1de1 at term 2
Nov 07 11:25:29 node01.k8s.com etcd: published {Name:etcd1 ClientURLs:} to cluster 42cecf80e3791d6c
Nov 07 11:25:29 node01.k8s.com etcd: ready to serve client requests
Nov 07 11:25:29 node01.k8s.com etcd: serving insecure client requests on 192.168.1.91:2379, this is strongly discouraged!
Nov 07 11:25:29 node01.k8s.com systemd: Started Etcd Server.
Nov 07 11:25:29 node01.k8s.com etcd: setting up the initial cluster version to 3.3
Nov 07 11:25:29 node01.k8s.com etcd: set the initial cluster version to 3.3
Nov 07 11:25:29 node01.k8s.com etcd: enabled capabilities for version 3.3

</code></pre>
<p>日志 <code>vim /var/log/messages</code> 表现如下:</p>
<pre><code>Nov7 11:25:27 node01 etcd: got bootstrap from DNS for etcd-server at 0=http://etcd3.k8s.com:2380
Nov7 11:25:27 node01 etcd: got bootstrap from DNS for etcd-server at 1=http://etcd2.k8s.com:2380
Nov7 11:25:27 node01 etcd: got bootstrap from DNS for etcd-server at etcd1=http://etcd1.k8s.com:2380
Nov7 11:25:27 node01 etcd: resolving etcd1.k8s.com:2380 to 192.168.1.91:2380
Nov7 11:25:27 node01 etcd: resolving etcd1.k8s.com:2380 to 192.168.1.91:2380
Nov7 11:25:28 node01 etcd: name = etcd1
Nov7 11:25:28 node01 etcd: data dir = /data/k8s/etcd/data
Nov7 11:25:28 node01 etcd: member dir = /data/k8s/etcd/data/member
Nov7 11:25:28 node01 etcd: dedicated WAL dir = /data/k8s/etcd/wal
Nov7 11:25:28 node01 etcd: heartbeat = 100ms
Nov7 11:25:28 node01 etcd: election = 1000ms
Nov7 11:25:28 node01 etcd: snapshot count = 100000
Nov7 11:25:28 node01 etcd: advertise client URLs = http://etcd1.k8s.com:2379,http://etcd1.k8s.com:4001
Nov7 11:25:28 node01 etcd: initial advertise peer URLs = http://etcd1.k8s.com:2380
Nov7 11:25:28 node01 etcd: initial cluster = 0=http://etcd3.k8s.com:2380,1=http://etcd2.k8s.com:2380,etcd1=http://etcd1.k8s.com:2380
</code></pre>
<p>测试：</p>
<pre><code># etcdctl --endpoints=http://192.168.1.91:2379 cluster-health
member 184beca37ca32d75 is healthy: got healthy result from http://etcd2.k8s.com:2379
member d79e9ae86b2a1de1 is healthy: got healthy result from http://etcd1.k8s.com:2379
member f7662e609b7e4013 is healthy: got healthy result from http://etcd3.k8s.com:2379
cluster is healthy
</code></pre>
<h1 id="etcd-tls动态集群基于dns的srv解析自动发现">ETCD TLS动态集群基于DNS的SRV解析自动发现</h1>
<blockquote>
<p>需要局域网内部有DNS服务器</p>
</blockquote>
<h2 id="添加srv解析-1">添加SRV解析</h2>
<blockquote>
<p>目前常用的内部DNS服务有两种，<code>bind</code>、<code>dnsmasq</code></p>
</blockquote>
<p><strong>在下面都会列出具体的配置，但只需要配置其中之一即可；</strong></p>
<h3 id="方法一-使用bind配置srv解析-1">方法一: 使用<code>bind</code>配置SRV解析</h3>
<p>如果内部没有<code>bind</code>服务，可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11806962.html</p>
<p>使用域名为 : <code>k8s.com</code>，在bind的zone文件中添加如下解析:</p>
<pre><code>etcd1   IN      A       192.168.1.91
etcd2   IN      A       192.168.1.92
etcd3   IN      A       192.168.1.93
_etcd-server-ssl._tcp.k8s.com.IN      SRV   10    10       2380   etcd1
_etcd-server-ssl._tcp.k8s.com.IN      SRV   10    10       2380   etcd2
_etcd-server-ssl._tcp.k8s.com.IN      SRV   10    10       2380   etcd3
_etcd-client-ssl._tcp.k8s.com.IN      SRV   10    10       2379   etcd1
_etcd-client-ssl._tcp.k8s.com.IN      SRV   10    10       2379   etcd2
_etcd-client-ssl._tcp.k8s.com.IN      SRV   10    10       2379   etcd3
</code></pre>
<p>修改之后重新加载配置文件:</p>
<pre><code># named-checkzone k8s.com k8s.com.zone
zone k8s.com/IN: loaded serial 0
OK
# rndc reload
server reload successful
</code></pre>
<h3 id="方法二-使用dnsmasq配置srv解析-1">方法二: 使用<code>dnsmasq</code>配置SRV解析</h3>
<p>如果内部没有<code>dnsmasq</code>服务，可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11809066.html</p>
<p>使用域名为 : <code>k8s.com</code>，具体修改如下:</p>
<p>在<code>/etc/dnsmasq_hosts</code>新增下面内容</p>
<pre><code>192.168.1.91 etcd1 etcd1.k8s.com
192.168.1.92 etcd2 etcd2.k8s.com
192.168.1.93 etcd3 etcd3.k8s.com
</code></pre>
<p>在 <code>/etc/dnsmasq.conf</code> 文件中增加下面SRV解析内容</p>
<pre><code>srv-host=_etcd-server-ssl._tcp.k8s.com,etcd1.k8s.com,2380,0,100
srv-host=_etcd-server-ssl._tcp.k8s.com,etcd2.k8s.com,2380,0,100
srv-host=_etcd-server-ssl._tcp.k8s.com,etcd3.k8s.com,2380,0,100
srv-host=_etcd-client-ssl._tcp.k8s.com,etcd1.k8s.com,2379,0,100
srv-host=_etcd-client-ssl._tcp.k8s.com,etcd2.k8s.com,2379,0,100
srv-host=_etcd-client-ssl._tcp.k8s.com,etcd3.k8s.com,2379,0,100
</code></pre>
<p>修改之后重启服务 <code>systemctl restart dnsmasq</code></p>
<h3 id="验证srv解析是否正常-1">验证SRV解析是否正常</h3>
<p><strong>查询SRV记录</strong></p>
<pre><code># dig @192.168.1.122 +noall +answer SRV _etcd-server-ssl._tcp.k8s.com
_etcd-server-ssl._tcp.k8s.com. 3600 IN        SRV        2380 0 100 etcd3.k8s.com.
_etcd-server-ssl._tcp.k8s.com. 3600 IN        SRV        2380 0 100 etcd2.k8s.com.
_etcd-server-ssl._tcp.k8s.com. 3600 IN        SRV        2380 0 100 etcd1.k8s.com.
</code></pre>
<p><strong>查询域名解析结果</strong></p>
<pre><code># dig @192.168.1.122 +noall +answer etcd1.k8s.com etcd2.k8s.com etcd3.k8s.com
etcd1.k8s.com.                86400        IN        A        192.168.1.91
etcd2.k8s.com.                86400        IN        A        192.168.1.92
etcd3.k8s.com.                86400        IN        A        192.168.1.93
</code></pre>
<h2 id="etcd-配置-1">ETCD 配置</h2>
<h3 id="node01-配置文件-3">node01 配置文件</h3>
<pre><code>ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="https://192.168.1.91:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.91:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd1"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd1.k8s.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd1.k8s.com:2379,https://etcd1.k8s.com:4001"
ETCD_DISCOVERY_SRV="k8s.com"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_PEER_AUTO_TLS="true"
</code></pre>
<h3 id="node02-配置文件-3">node02 配置文件</h3>
<pre><code>ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="https://192.168.1.92:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.92:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd2"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd2.k8s.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd2.k8s.com:2379"
ETCD_DISCOVERY_SRV="k8s.com"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_PEER_AUTO_TLS="true"
</code></pre>
<h3 id="node03-配置文件-3">node03 配置文件</h3>
<pre><code>ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="https://192.168.1.93:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.93:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd3"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd3.k8s.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd3.k8s.com:2379"
ETCD_DISCOVERY_SRV="k8s.com"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_PEER_AUTO_TLS="true"
</code></pre>
<h2 id="启动测试-2">启动测试</h2>
<p>启动</p>
<pre><code># systemctl restart etcd
# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-11-07 12:38:37 CST; 4s ago
Main PID: 13460 (etcd)
    Tasks: 8
   Memory: 16.6M
   CGroup: /system.slice/etcd.service
         └─13460 /usr/bin/etcd --name=etcd3 --data-dir=/data/k8s/etcd/data --listen-client-urls=https://192.168.1.93:2379

Nov 07 12:38:36 node03.k8s.com etcd: established a TCP streaming connection with peer 40a8f19a5db99534 (stream Message writer)
Nov 07 12:38:36 node03.k8s.com etcd: established a TCP streaming connection with peer 40a8f19a5db99534 (stream MsgApp v2 writer)
Nov 07 12:38:37 node03.k8s.com etcd: 9888555207dbf0e0 received a MsgVote message with higher term from a0d541999e9eb3b3
Nov 07 12:38:37 node03.k8s.com etcd: 9888555207dbf0e0 became follower at term 98
Nov 07 12:38:37 node03.k8s.com etcd: 9888555207dbf0e0 cast MsgVote for a0d541999e9eb3b3 at term 98
Nov 07 12:38:37 node03.k8s.com etcd: raft.node: 9888555207dbf0e0 elected leader a0d541999e9eb3b3 at term 98
Nov 07 12:38:37 node03.k8s.com etcd: published {Name:etcd3 ClientURLs:} to cluster f445a02ce3dc6a02
Nov 07 12:38:37 node03.k8s.com etcd: ready to serve client requests
Nov 07 12:38:37 node03.k8s.com etcd: serving client requests on 192.168.1.93:2379
Nov 07 12:38:37 node03.k8s.com systemd: Started Etcd Server.

</code></pre>
<p><strong>日志体现</strong></p>
<pre><code>Nov7 12:38:36 node01 etcd: added member 40a8f19a5db99534 to cluster f445a02ce3dc6a02
Nov7 12:38:36 node01 etcd: starting peer 40a8f19a5db99534...
Nov7 12:38:36 node01 etcd: started HTTP pipelining with peer 40a8f19a5db99534
Nov7 12:38:36 node01 etcd: started streaming with peer 40a8f19a5db99534 (writer)
Nov7 12:38:36 node01 etcd: started peer 40a8f19a5db99534
Nov7 12:38:36 node01 etcd: added peer 40a8f19a5db99534
Nov7 12:38:36 node01 etcd: added member 9888555207dbf0e0 to cluster f445a02ce3dc6a02
Nov7 12:38:36 node01 etcd: starting peer 9888555207dbf0e0...
Nov7 12:38:36 node01 etcd: started HTTP pipelining with peer 9888555207dbf0e0
Nov7 12:38:36 node01 etcd: started peer 9888555207dbf0e0
Nov7 12:38:36 node01 etcd: added peer 9888555207dbf0e0
Nov7 12:38:36 node01 etcd: added member a0d541999e9eb3b3 to cluster f445a02ce3dc6a02
</code></pre>
<p>测试集群状态：</p>
<pre><code>ETCDCTL_API=3 etcdctl --endpoints=https://etcd1.k8s.com:2379,https://etcd2.k8s.com:2379,https://etcd3.k8s.com:2379 \
    --cacert=/etc/kubernetes/cert/ca.pem \
    --cert=/etc/etcd/cert/etcd.pem \
    --key=/etc/etcd/cert/etcd-key.pem endpoint health

# 输出
https://etcd1.k8s.com:2379 is healthy: successfully committed proposal: took = 4.269468ms
https://etcd3.k8s.com:2379 is healthy: successfully committed proposal: took = 1.58797ms
https://etcd2.k8s.com:2379 is healthy: successfully committed proposal: took = 1.622151ms
</code></pre>
<pre><code>etcdctl \
--endpoint=https://etcd1.k8s.com:2379 \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/etcd/cert/etcd.pem \
--key-file=/etc/etcd/cert/etcd-key.pem cluster-health

# 输出
member 40a8f19a5db99534 is healthy: got healthy result from https://etcd2.k8s.com:2379
member 9888555207dbf0e0 is healthy: got healthy result from https://etcd3.k8s.com:2379
member a0d541999e9eb3b3 is healthy: got healthy result from https://etcd1.k8s.com:2379
cluster is healthy
</code></pre>
<h1 id="报错解决">报错解决</h1>
<h2 id="1-证书报错-bad-certificate">1. 证书报错 bad certificate</h2>
<p><strong>日志中报错:</strong></p>
<pre><code>Nov7 12:37:03 node01 etcd: rejected connection from "192.168.1.93:46294" (error "remote error: tls: bad certificate", ServerName "k8s.com")
</code></pre>
<h3 id="解决">解决</h3>
<p>报错的意思是在生成ETCD的TLS证书的时候，没有把对应的域名加进去</p>
<p><strong>在创建ETCD的TLS证书请求的文件中加入对应的域名</strong></p>
<pre><code>cd /opt/k8s/work
cat &gt; etcd-csr.json &lt;&lt;EOF
{
"CN": "etcd",
"hosts": [
    "127.0.0.1",
    "192.168.1.91",
    "192.168.1.92",
    "192.168.1.93",
    "k8s.com",          # 这里的域名查看是否正确
    "etcd1.k8s.com",
    "etcd2.k8s.com",
    "etcd3.k8s.com"
],
"key": {
    "algo": "rsa",
    "size": 2048
},
"names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
]
}
EOF
</code></pre>
<h2 id="2-dns-的-srv-解析报错-cannot-find-local-etcd-member-etcd1-in-srv-records">2. <code>DNS</code> 的 <code>SRV</code> 解析报错 cannot find local etcd member "etcd1" in SRV records</h2>
<p><strong>DNS 如果配置有问题，会有如下报错:</strong></p>
<pre><code>etcd: error setting up initial cluster: cannot find local etcd member "etcd1" in SRV records
</code></pre>
<p>这里是表示DNS在配置SRV解析的时候报错，请仔细查看解析配置:</p>
<p>SRV解析分为两种，一种是http不带证书的解析，一种是https带证书的解析，是有区别的，如果配置错误就会包上述错误</p>
<h3 id="http不带证书解析如下">http不带证书解析如下</h3>
<h4 id="bind-的解析">bind 的解析</h4>
<p>编辑 <code>/var/named/k8s.com.zone</code> 文件</p>
<pre><code>etcd1   IN      A       192.168.1.91
etcd2   IN      A       192.168.1.92
etcd3   IN      A       192.168.1.93
_etcd-server._tcp.k8s.com.IN      SRV   10    10       2380   etcd1
_etcd-server._tcp.k8s.com.IN      SRV   10    10       2380   etcd2
_etcd-server._tcp.k8s.com.IN      SRV   10    10       2380   etcd3
_etcd-client._tcp.k8s.com.IN      SRV   10    10       2379   etcd1
_etcd-client._tcp.k8s.com.IN      SRV   10    10       2379   etcd2
_etcd-client._tcp.k8s.com.IN      SRV   10    10       2379   etcd3
</code></pre>
<h4 id="dnsmasq-的解析">dnsmasq 的解析</h4>
<p>在<code>/etc/dnsmasq_hosts</code>新增下面内容</p>
<pre><code>192.168.1.91 etcd1 etcd1.k8s.com
192.168.1.92 etcd2 etcd2.k8s.com
192.168.1.93 etcd3 etcd3.k8s.com
</code></pre>
<p>在 <code>/etc/dnsmasq.conf</code> 文件中增加下面SRV解析内容</p>
<pre><code>srv-host=_etcd-server._tcp.k8s.com,etcd1.k8s.com,2380,0,100
srv-host=_etcd-server._tcp.k8s.com,etcd2.k8s.com,2380,0,100
srv-host=_etcd-server._tcp.k8s.com,etcd3.k8s.com,2380,0,100
srv-host=_etcd-client._tcp.k8s.com,etcd1.k8s.com,2380,0,100
srv-host=_etcd-client._tcp.k8s.com,etcd2.k8s.com,2380,0,100
srv-host=_etcd-client._tcp.k8s.com,etcd3.k8s.com,2380,0,100
</code></pre>
<h3 id="https带证书解析如下">https带证书解析如下</h3>
<h4 id="bind-的解析-1">bind 的解析</h4>
<p>编辑 <code>/var/named/k8s.com.zone</code> 文件</p>
<pre><code>etcd1   IN      A       192.168.1.91
etcd2   IN      A       192.168.1.92
etcd3   IN      A       192.168.1.93
_etcd-server-ssl._tcp.k8s.com.IN      SRV   10    10       2380   etcd1
_etcd-server-ssl._tcp.k8s.com.IN      SRV   10    10       2380   etcd2
_etcd-server-ssl._tcp.k8s.com.IN      SRV   10    10       2380   etcd3
_etcd-client-ssl._tcp.k8s.com.IN      SRV   10    10       2379   etcd1
_etcd-client-ssl._tcp.k8s.com.IN      SRV   10    10       2379   etcd2
_etcd-client-ssl._tcp.k8s.com.IN      SRV   10    10       2379   etcd3
</code></pre>
<h4 id="dnsmasq-的解析-1">dnsmasq 的解析</h4>
<p>在<code>/etc/dnsmasq_hosts</code>新增下面内容</p>
<pre><code>192.168.1.91 etcd1 etcd1.k8s.com
192.168.1.92 etcd2 etcd2.k8s.com
192.168.1.93 etcd3 etcd3.k8s.com
</code></pre>
<p>在 <code>/etc/dnsmasq.conf</code> 文件中增加下面SRV解析内容</p>
<pre><code>srv-host=_etcd-server-ssl._tcp.k8s.com,etcd1.k8s.com,2380,0,100
srv-host=_etcd-server-ssl._tcp.k8s.com,etcd2.k8s.com,2380,0,100
srv-host=_etcd-server-ssl._tcp.k8s.com,etcd3.k8s.com,2380,0,100
srv-host=_etcd-client-ssl._tcp.k8s.com,etcd1.k8s.com,2379,0,100
srv-host=_etcd-client-ssl._tcp.k8s.com,etcd2.k8s.com,2379,0,100
srv-host=_etcd-client-ssl._tcp.k8s.com,etcd3.k8s.com,2379,0,100
</code></pre>


</div>
<div id="MySignature" role="contentinfo">
    技术男一枚，喜欢做技术分享，把学习的过程，以及遇到问题的解决过程都愿意分享给大家，博客中如有不足，请留言或者联系博主，感谢。
邮箱: sijiayong000@163.com
Q Q: 601566386<br><br>
来源：https://www.cnblogs.com/winstom/p/11811373.html
頁: [1]
查看完整版本: CentOS 7 ETCD集群配置大全