kubernetes集群部署高可用Postgresql的Stolon方案
<h2 id="目录">目录</h2><p>前言</p>
<h2 id="前言">....前言</h2>
<p><strong>本文选用Stolon的方式搭建Postgresql高可用方案，主要为Harbor提供高可用数据库，Harbor搭建可查看kubernetes搭建Harbor无坑及Harbor仓库同步，之后会提供redis高可用及Harbor高可用方案搭建</strong></p>
<h2 id="方案比较">方案比较</h2>
<p><strong>几种postgresql高可用方案简单比较：</strong></p>
<blockquote>
<p>引用https://studygolang.com/articles/19002?fr=sidebar</p>
<ul>
<li>首先repmgr这种方案的算法有明显缺陷，非主流分布式算法，直接pass；</li>
<li>Stolon和Patroni相对于Crunchy更加Cloud Native， 后者是基于pgPool实现。</li>
<li>Crunchy和Patroni相对于Stolon有更多的使用者，并且提供了Operator对于以后的管理和扩容</li>
</ul>
</blockquote>
<p>根据上面简单的比较，最终选择的stolon，作者选择的是Patroni，感觉实际区别并不大。</p>
<h2 id="一stolon概述">一、Stolon概述：</h2>
<p>Stolon（https://github.com/sorintlab/stolon<br>
）<br>
是由3个部分组成的：</p>
<ul>
<li>keeper：他负责管理PostgreSQL的实例汇聚到由sentinel(s)提供的clusterview。</li>
<li>sentinel：it负责发现并且监控keeper，并且计算最理想的clusterview。</li>
<li>proxy：客户端的接入点。它强制连接到右边PostgreSQL的master并且强制关闭连接到由非选举产生的master。<br>
Stolon 用etcd或者consul作为主要的集群状态存储。</li>
</ul>
<h2 id="二installation">二、Installation</h2>
<pre><code>git clone https://github.com/sorintlab/stolon.git
cd XXX/stolon/examples/kubernetes
</code></pre>
<p>如图所示<br>
<img src="https://upload-images.jianshu.io/upload_images/9371663-6ea308fa56caf09c.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="stolon" loading="lazy"><br>
如有兴趣可查看官网搭建：https://github.com/sorintlab/stolon/blob/master/examples/kubernetes/README.md<br>
<strong>如下为yaml中注意修改的地方</strong></p>
<ul>
<li><strong>stolon-keeper.yaml 中设置Postgresql用户名</strong></li>
</ul>
<pre><code>- name: STKEEPER_PG_SU_USERNAME
value: "postgres"
</code></pre>
<ul>
<li><strong>stolon-keeper.yaml 中设置stolon挂载卷</strong></li>
</ul>
<pre><code> volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "512Mi"
storageClassName: nfs
</code></pre>
<ul>
<li><strong>secret.yaml中设置用户密码</strong></li>
</ul>
<pre><code>apiVersion: v1
kind: Secret
metadata:
name: stolon
type: Opaque
data:
##echo -n "password1" | base64
password: cGFzc3dvcmQx
</code></pre>
<p><strong>如下是作者整理的完整的stolon的编排文件，可直接修改使用</strong></p>
<pre><code># This is an example and generic rbac role definition for stolon. It could be
# fine tuned and split per component.
# The required permission per component should be:
# keeper/proxy/sentinel: update their own pod annotations
# sentinel/stolonctl: get, create, update configmaps
# sentinel/stolonctl: list components pods
# sentinel/stolonctl: get components pods annotations
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: stolon
namespace: default
rules:
- apiGroups:
- ""
resources:
- pods
- configmaps
- events
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: stolon
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: stolon
subjects:
- kind: ServiceAccount
name: default
namespace: default
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: stolon-sentinel
spec:
replicas: 2
template:
metadata:
labels:
component: stolon-sentinel
stolon-cluster: kube-stolon
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
spec:
containers:
- name: stolon-sentinel
image: sorintlab/stolon:master-pg10
command:
- "/bin/bash"
- "-ec"
- |
exec gosu stolon stolon-sentinel
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: STSENTINEL_CLUSTER_NAME
valueFrom:
fieldRef:
fieldPath: metadata.labels['stolon-cluster']
- name: STSENTINEL_STORE_BACKEND
value: "kubernetes"
- name: STSENTINEL_KUBE_RESOURCE_KIND
value: "configmap"
- name: STSENTINEL_METRICS_LISTEN_ADDRESS
value: "0.0.0.0:8080"
## Uncomment this to enable debug logs
#- name: STSENTINEL_DEBUG
#value: "true"
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Secret
metadata:
name: stolon
type: Opaque
data:
password: cGFzc3dvcmQx
---
# PetSet was renamed to StatefulSet in k8s 1.5
# apiVersion: apps/v1alpha1
# kind: PetSet
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: stolon-keeper
spec:
serviceName: "stolon-keeper"
replicas: 2
template:
metadata:
labels:
component: stolon-keeper
stolon-cluster: kube-stolon
annotations:
pod.alpha.kubernetes.io/initialized: "true"
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
spec:
terminationGracePeriodSeconds: 10
containers:
- name: stolon-keeper
image: sorintlab/stolon:master-pg10
command:
- "/bin/bash"
- "-ec"
- |
# Generate our keeper uid using the pod index
IFS='-' read -ra ADDR <<< "$(hostname)"
export STKEEPER_UID="keeper${ADDR[-1]}"
export POD_IP=$(hostname -i)
export STKEEPER_PG_LISTEN_ADDRESS=$POD_IP
export STOLON_DATA=/stolon-data
chown stolon:stolon $STOLON_DATA
exec gosu stolon stolon-keeper --data-dir $STOLON_DATA
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: STKEEPER_CLUSTER_NAME
valueFrom:
fieldRef:
fieldPath: metadata.labels['stolon-cluster']
- name: STKEEPER_STORE_BACKEND
value: "kubernetes"
- name: STKEEPER_KUBE_RESOURCE_KIND
value: "configmap"
- name: STKEEPER_PG_REPL_USERNAME
value: "repluser"
# Or use a password file like in the below supersuser password
- name: STKEEPER_PG_REPL_PASSWORD
value: "replpassword"
- name: STKEEPER_PG_SU_USERNAME
value: "postgres"
- name: STKEEPER_PG_SU_PASSWORDFILE
value: "/etc/secrets/stolon/password"
- name: STKEEPER_METRICS_LISTEN_ADDRESS
value: "0.0.0.0:8080"
# Uncomment this to enable debug logs
#- name: STKEEPER_DEBUG
#value: "true"
ports:
- containerPort: 5432
- containerPort: 8080
volumeMounts:
- mountPath: /stolon-data
name: data
- mountPath: /etc/secrets/stolon
name: stolon
volumes:
- name: stolon
secret:
secretName: stolon
# Define your own volumeClaimTemplate. This example uses dynamic PV provisioning with a storage class named "standard" (so it will works by default with minikube)
# In production you should use your own defined storage-class and configure your persistent volumes (statically or dynamically using a provisioner, see related k8s doc).
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "512Mi"
storageClassName: nfs
---
apiVersion: v1
kind: Service
metadata:
name: stolon-proxy-service
spec:
ports:
- port: 5432
targetPort: 5432
selector:
component: stolon-proxy
stolon-cluster: kube-stolon
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: stolon-proxy
spec:
replicas: 2
template:
metadata:
labels:
component: stolon-proxy
stolon-cluster: kube-stolon
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
spec:
containers:
- name: stolon-proxy
image: sorintlab/stolon:master-pg10
command:
- "/bin/bash"
- "-ec"
- |
exec gosu stolon stolon-proxy
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: STPROXY_CLUSTER_NAME
valueFrom:
fieldRef:
fieldPath: metadata.labels['stolon-cluster']
- name: STPROXY_STORE_BACKEND
value: "kubernetes"
- name: STPROXY_KUBE_RESOURCE_KIND
value: "configmap"
- name: STPROXY_LISTEN_ADDRESS
value: "0.0.0.0"
- name: STPROXY_METRICS_LISTEN_ADDRESS
value: "0.0.0.0:8080"
## Uncomment this to enable debug logs
#- name: STPROXY_DEBUG
#value: "true"
ports:
- containerPort: 5432
- containerPort: 8080
readinessProbe:
tcpSocket:
port: 5432
initialDelaySeconds: 10
timeoutSeconds: 5
</code></pre>
<h2 id="三部署stolon">三、部署stolon</h2>
<pre><code>kubectl applay -f stolon.yaml
</code></pre>
<blockquote>
<h3 id="initialize-the-cluster大概意思是stolon初始化k8s集群可以大概看下官网解释">Initialize the cluster（大概意思是stolon初始化k8s集群，可以大概看下官网解释）</h3>
<p>All the stolon components wait for an existing clusterdata entry in the store. So the first time you have to initialize a new cluster. For more details see the cluster initialization doc. You can do this step at every moment, now or after having started the stolon components.<br>
You can execute stolonctl in different ways:</p>
<ul>
<li>as a one shot command executed inside a temporary pod:</li>
</ul>
<pre><code>kubectl run -i -t stolonctl --image=sorintlab/stolon:master-pg10 --restart=Never --rm -- /usr/local/bin/stolonctl --cluster-name=kube-stolon --store-backend=kubernetes --kube-resource-kind=configmap init
</code></pre>
<ul>
<li>from a machine that can access the store backend:</li>
</ul>
<pre><code>stolonctl --cluster-name=kube-stolon --store-backend=kubernetes --kube-resource-kind=configmap init
</code></pre>
<ul>
<li>later from one of the pods running the stolon components.</li>
</ul>
</blockquote>
<h4 id="执行">执行</h4>
<pre><code>kubectl run -i -t stolonctl --image=sorintlab/stolon:master-pg10 --restart=Never --rm -- /usr/local/bin/stolonctl --cluster-name=kube-stolon --store-backend=kubernetes --kube-resource-kind=configmap init
</code></pre>
<h4 id="如图所示部署成功">如图所示，部署成功</h4>
<p><img src="https://upload-images.jianshu.io/upload_images/9371663-69f109a5e1e2282c.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240" alt="stolon pod" loading="lazy"></p>
<h2 id="四卸载postgresql数据库">四、卸载Postgresql数据库</h2>
<pre><code>kubectl delete -f stolon.yaml
kubectl delete pvc data-stolon-keeper-0 data-stolon-keeper-1
</code></pre>
<h2 id="五验证postgresql安装成功也可简单测试下">五、验证Postgresql安装成功（也可简单测试下）</h2>
<h4 id="1验证数据同步">1、验证数据同步</h4>
<p>连接master并且建立test表</p>
<blockquote>
<p>psql --host <ip> --port 30543 postgres -U stolon -W<br>
postgres=# create table test (id int primary key not null,<br>
value text not null);<br>
CREATE TABLE<br>
postgres=# insert into test values (1, 'value1');<br>
INSERT 0 1<br>
postgres=# select * from test;<br>
id | value<br>
---- -------- <br>
1 | value1<br>
(1 row)</ip></p>
</blockquote>
<p>也可进入Pod执行postgresql命令</p>
<pre><code>kubectl exec -ti stolon-proxy-5977cdbcfc-csnkq bash
#登入sql
psql --host localhost --port 5432 postgres -U postgres
\l #列出所有数据库
\c dbname #切换数据库
CREATE TABLE
insert into test values (1, 'value1');
INSERT 0 1
select * from test;
\d #列出当前数据库的所有表
\q #退出数据库
</code></pre>
<p>连接slave并且检查数据。你可以写一些信息以便确认请求已经被slave处理了。</p>
<blockquote>
<p>psql --host <ip> --port 30544 postgres -U stolon -W<br>
postgres=# select * from test;<br>
id | value<br>
---- -------- <br>
1 | value1<br>
(1 row)</ip></p>
</blockquote>
<h4 id="2测试failover">2、测试failover</h4>
<p>这个案例是官方代码库中statefullset的一个例子。<br>
简单的说，就是为模拟了master挂掉，我们先删除了master的statefulset又删除了master的pod。</p>
<pre><code>kubectl delete statefulset stolon-keeper --cascade=false
kubectl delete pod stolon-keeper-0
</code></pre>
<p>然后，在sentinel的log中我们可以看到新的master被选举出来了。</p>
<blockquote>
<p>no keeper info available db=cb96f42d keeper=keeper0<br>
no keeper info available db=cb96f42d keeper=keeper0<br>
master db is failed db=cb96f42d keeper=keeper0<br>
trying to find a standby to replace failed master<br>
electing db as the new master db=087ce88a keeper=keeper1</p>
</blockquote>
<p>现在，在刚才的那两个终端中如果我们重复上一个命令，我们可以看到如下输出。</p>
<blockquote>
<p>postgres=# select * from test;<br>
server closed the connection unexpectedly<br>
This probably means the server terminated abnormally<br>
before or while processing the request.<br>
The connection to the server was lost. Attempting reset:<br>
Succeeded.<br>
postgres=# select * from test;<br>
id | value<br>
---- -------- <br>
1 | value1<br>
(1 row)</p>
</blockquote>
<p>Kubernetes的service把不可用的pod去掉，把请求转到可用的pod上。所以新的读取连接被路由到了健康的pod上。</p>
<h4 id="也可用chaoskube模拟随机的pod挂掉准生产可以测试下">.也可用chaoskube模拟随机的pod挂掉（准生产可以测试下）</h4>
<p>另一个测试集群弹性（resilience）的好方法是用chaoskube。Chaoskube是一个小的服务程序，它可以周期性的在集群里随机的kill掉一些的pod。它也可以用helm charts部署。</p>
<pre><code>helm install --set labels="release=factualcrocodile,
component!=factual-crocodine-etcd" --set
interval=5m stable/chaoskube
</code></pre>
<p>这条命令会运行chaoskube，它会每5分钟删除一个pod。它会选择label中release=factual-crocodile的pod，但是会忽略etcd的pod。</p>
<h3 id="本文按照官网搭建主要为之后的harbor高可用做准备有情趣的伙伴点个赞之后会续写redisharbor高可用">本文按照官网搭建，主要为之后的Harbor高可用做准备，有情趣的伙伴点个赞，之后会续写redis、Harbor高可用</h3>
<p>参考资料：<br>
https://my.oschina.net/u/2306127/blog/2991474<br>
https://github.com/sorintlab/stolon/tree/master/examples/kubernetes<br>
https://studygolang.com/articles/19002?fr=sidebar</p><br><br>
来源：https://www.cnblogs.com/keep-live/p/11543770.html
頁:
[1]