Kubernetes之动态Jenkins slave

林深叶茂 發表於 2019-8-29 16:42:00

<h3 id="一前提">一、前提</h3>
本次实践前，需已完成以下过程：
1、搭建好一个Kubernetes集群(本实践为单节点集群)，网上参考较多，不赘述。
2、选取kubernetes集群外的一台服务器安装 NFS服务端，并在集群内每个节点安装 NFS客户端; 
NFS服务端所在的服务器IP为 <code>10.141.211.178</code>，记为 nfs server；而集群master服务器，记为 master; 
(1) nfs server创建存储目录 <code>/data/k8s/</code>并执行命令：<code>chmod 755 /data/k8s/</code>, 并关闭防火墙 
(2) nfs server安装NFS，执行: <code>yum -y install nfs-utils rpcbind</code>， 
再配置NFS，执行：<code>vi /etc/exports</code>，在该文件内添加内容：<code>/data/k8s*(rw,sync,no_root_squash)</code>， 
然后启动NFS服务，执行：
<pre><code class="language-shell"> # systemctl start rpcbind && systemctl enable rpcbind
# systemctl start nfs && systemctl enable nfs
</code></pre>
 (3) master同样安装并启动NFS，执行：
<pre><code class="language-shell"> # yum -y install nfs-utils rpcbind
# systemctl start rpcbind && systemctl enable rpcbind
# systemctl start nfs && systemctl enable nfs
</code></pre>
 再执行：<code>showmount -e 10.141.211.178</code>，可看到共享目录 /data/k8s
<h3 id="二集群安装jenkins">二、集群安装Jenkins</h3>
Jenkins master的安装，需要将数据持久化。可以利用NFS作为存储资源，创建PVC对象来挂载。PV/PVC配置文件pvc.yaml如下：
<pre><code class="language-pvc.yaml">apiVersion: v1
kind: PersistentVolume
metadata:
name: opspv
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Delete
nfs:
server: 10.141.211.178 #注意：此处为NFS服务器的地址
path: /data/k8s

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: opspvc
namespace: kube-ops
spec:
accessModes:
- ReadWriteMany
resources:
requests:
 storage: 20Gi
</code></pre>
同时，对于即将创建的Jenkins master资源对象，需要授予其一些权限，比如增删改查等。相应的配置文件rbac.yaml如下：
<pre><code class="language-rbac.yaml">apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: kube-ops

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins
rules:
- apiGroups: ["extensions", "apps"]
resources: ["deployments"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: jenkins
namespace: kube-ops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
namespace: kube-ops
</code></pre>
然后，基于jenkins/jenkins:lts 镜像创建jenkins master镜像，配置文件jenkins.yaml 如下：
<pre><code class="language-jenkins.yaml">apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jenkins
namespace: kube-ops
spec:
template:
metadata:
 labels:
 app: jenkins
spec:
 terminationGracePeriodSeconds: 10
 serviceAccount: jenkins
 containers:
 - name: jenkins
 image: jenkins/jenkins:lts
 imagePullPolicy: IfNotPresent
 ports:
 - containerPort: 8080
 name: web
 protocol: TCP
 - containerPort: 50000
 name: agent
 protocol: TCP
 resources:
 limits:
 cpu: 1000m
 memory: 1Gi
 requests:
 cpu: 500m
 memory: 512Mi
 livenessProbe:
 httpGet:
 path: /login
 port: 8080
 initialDelaySeconds: 60
 timeoutSeconds: 5
 failureThreshold: 12
 readinessProbe:
 httpGet:
 path: /login
 port: 8080
 initialDelaySeconds: 60
 timeoutSeconds: 5
 failureThreshold: 12
 volumeMounts:
 - name: jenkinshome
 subPath: jenkins
 mountPath: /var/jenkins_home
 env:
 - name: LIMITS_MEMORY
 valueFrom:
 resourceFieldRef:
 resource: limits.memory
 divisor: 1Mi
 - name: JAVA_OPTS
 value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
 securityContext:
 fsGroup: 1000
 volumes:
 - name: jenkinshome
 persistentVolumeClaim:
 claimName: opspvc

---
apiVersion: v1
kind: Service
metadata:
name: jenkins
namespace: kube-ops
labels:
app: jenkins
spec:
selector:
app: jenkins
type: NodePort
ports:
- name: web
port: 8080
targetPort: web
nodePort: 30002
- name: agent
port: 50000
targetPort: agent
</code></pre>
最后，在一个目录内分别创建以上3个文件，执行命令如下：
<pre><code># kubectl create namespace kube-ops
# kubectl create -f pvc.yaml
# kubectl create -f rbac.yaml
# kubectl create -f jenkins.yaml(此步执行会出现文件权限问题，解决办法为：
 先在nfs server服务器执行：chown -R 1000 /data/k8s/jenkins
 然后在master执行：kubectl delete -f jenkins.yaml
 kubectl create -f jenkins.yaml)
</code></pre>
此时，我们通过命令<code>kubectl -n kube-ops get pod</code>可以查看到jenkins已成功创建。
<h3 id="三jenkins配置动态slave">三、Jenkins配置动态slave</h3>
<h4 id="1初始化jenkins配置">1、初始化Jenkins配置</h4>
浏览器打开<code>masterIP:30002</code>，如下： 
<img src="https://img2018.cnblogs.com/blog/1389290/201908/1389290-20190829161323801-797531597.jpg" alt="" loading="lazy">
其中的管理员密码，我们既可以进入容器内对应的目录查看，也可以在nfs server服务器上执行命令： 
<code>cat /data/k8s/jenkins/secrets/initialAdminPassword</code>来查看；然后选择安装推荐的插件，如下： 
<img src="https://img2018.cnblogs.com/blog/1389290/201908/1389290-20190829161708089-606300528.png" alt="" loading="lazy">
然后添加管理员账户即可进入Jenkins界面。
<h4 id="2配置jenkins-slave">2、配置jenkins slave</h4>
(1) 安装Kubernetes插件 
进入 Manage Jenkins—>Manage Plugins—>可选插件(Available)—>Kubernetes plugin勾选，直接安装即可。 
(2) 配置Kubernetes插件 
点击Manage Jenkins—>Configure System—>云—>新增一个云—>Kubernetes，如下： 
<img src="https://img2018.cnblogs.com/blog/1389290/201908/1389290-20190829161901489-1319929149.png" alt="" loading="lazy">
然后配置如下： 
先注意名称默认为<code>kubernetes</code>，然后 Kubernetes地址填写<code>https://kubernetes.default.svc.cluster.local</code>，命名空间为<code>kube-ops</code>；接着点击右边的连接测试按钮，如果显示<code>Connection test successful</code>，表示Jenkins可以和Kubernetes集群正常通信了。最后，在Jenkins地址，填入：<code>http://服务名.kube-ops.svc.cluster.local:8080</code>，如下所示： 
<img src="https://img2018.cnblogs.com/blog/1389290/201908/1389290-20190829161918179-2071956276.png" alt="" loading="lazy">
(3) 配置 Kubernetes Pod Template 
关于 Kubernetes Pod Template部分的配置，其实就是对jenkins slave的配置。具体配置如下： 
<img src="https://img2018.cnblogs.com/blog/1389290/201908/1389290-20190829163619198-2140572185.png" alt="" loading="lazy">
图中标记的地方较为重要，不要填错。其中标签列表部分后面仍有用到；Docker 镜像部分，是本人基于 cnych/jenkins:jnlp6 镜像基础上继续定制的镜像，包含maven、docker、docker-compose、kubectl等工具。 
另外，添加卷如下： 
<img src="https://img2018.cnblogs.com/blog/1389290/201908/1389290-20190829162048505-744311114.png" alt="" loading="lazy">
添加这两个 Host Path Volume，是为了更好地在jenkins slave容器中使用docker 和 kubectl 工具，所以挂载了宿主机的部分目录。 
然后，设置Service Account如下： 
<img src="https://img2018.cnblogs.com/blog/1389290/201908/1389290-20190829162102850-683419121.png" alt="" loading="lazy">
最后，点击保存即可。
<h4 id="3测试jenkins-slave">3.测试jenkins slave</h4>
首先新建一个名为test 的 Freestyle project 项目，其配置如下： 
<img src="https://img2018.cnblogs.com/blog/1389290/201908/1389290-20190829162119078-1046645082.png" alt="" loading="lazy">
这里的标签表达式，正是Kubernetes Pod Template的标签列表的内容。 
然后，增加构建步骤—>执行shell，如下: 
<img src="https://img2018.cnblogs.com/blog/1389290/201908/1389290-20190829162210019-557626074.png" alt="" loading="lazy">
具体shell如下：
<pre><code class="language-shell">echo "测试 Kubernetes 动态生成 jenkins slave"

echo "===========mvn==========="
mvn --version
echo $PATH

echo "==============docker in docker==========="
which docker
docker version

echo "==============docker-compose==========="
docker-compose version

echo "=============kubectl============="
kubectl get pods
</code></pre>
保存之后，点击立即构建。控制台输出如下：
<pre><code>Started by user admin
Running as SYSTEM
Agent jnlp-slave-pk06f is provisioned from template Kubernetes Pod Template
---
apiVersion: "v1"
kind: "Pod"
metadata:
annotations: {}
labels:
jenkins: "slave"
jenkins/jnlp-slave: "true"
name: "jnlp-slave-pk06f"
spec:
containers:
- env:
- name: "JENKINS_SECRET"
 value: "********"
- name: "JENKINS_AGENT_NAME"
 value: "jnlp-slave-pk06f"
- name: "JENKINS_NAME"
 value: "jnlp-slave-pk06f"
- name: "JENKINS_AGENT_WORKDIR"
 value: "/home/jenkins/agent"
- name: "JENKINS_URL"
 value: "http://jenkins.kube-ops.svc.cluster.local:8080/"
image: "zhongyuanzhao000/jenkins-slave:jnlp"
imagePullPolicy: "IfNotPresent"
name: "jnlp"
resources:
 limits: {}
 requests: {}
securityContext:
 privileged: false
tty: true
volumeMounts:
- mountPath: "/var/run/docker.sock"
 name: "volume-0"
 readOnly: false
- mountPath: "/root/.kube"
 name: "volume-1"
 readOnly: false
- mountPath: "/home/jenkins/agent"
 name: "workspace-volume"
 readOnly: false
workingDir: "/home/jenkins/agent"
nodeSelector: {}
restartPolicy: "Never"
serviceAccount: "jenkins"
volumes:
- hostPath:
 path: "/var/run/docker.sock"
name: "volume-0"
- hostPath:
 path: "/root/.kube"
name: "volume-1"
- emptyDir:
 medium: ""
name: "workspace-volume"

Building remotely on jnlp-slave-pk06f (jnlp-slave) in workspace /home/jenkins/agent/workspace/test
$ /bin/sh -xe /tmp/jenkins3820575614440094591.sh
+ echo 测试 Kubernetes 动态生成 jenkins slave
测试 Kubernetes 动态生成 jenkins slave
+ echo ===========mvn===========
===========mvn===========
+ mvn --version
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00)
Maven home: /usr/local
Java version: 1.8.0_212, vendor: Oracle Corporation, runtime: /usr/local/openjdk-8/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-327.el7.x86_64", arch: "amd64", family: "unix"
+ echo /usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ echo ==============docker in docker===========
==============docker in docker===========
+ which docker
/usr/local/bin/docker
+ docker version
Client: Docker Engine - Community
Version: 18.09.8
API version: 1.39
Go version: go1.10.8
Git commit: 0dd43dd87f
Built: Wed Jul 17 17:38:58 2019
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 18.09.7
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 2d0083d
Built: Thu Jun 27 17:26:28 2019
OS/Arch: linux/amd64
Experimental: false
+ echo ==============docker-compose===========
==============docker-compose===========
+ docker-compose version
docker-compose version 1.23.2, build 1110ad01
docker-py version: 3.6.0
CPython version: 3.6.7
OpenSSL version: OpenSSL 1.1.0f25 May 2017
+ echo =============kubectl=============
=============kubectl=============
+ kubectl get pods
NAME READY STATUS RESTARTS AGE
jenkins-575b84fb7b-59s5h 1/1 Running 0 29h
jnlp-slave-pk06f 1/1 Running 0 7s
Finished: SUCCESS
</code></pre>
其中，你可以发现jenkins创建了jnlp-slave-pk06f的slave对象；而当该任务执行完之后，你在master上获取pod就会发现 jnlp-slave-pk06f 这个slave自动消失了，这就是动态jenkins slave的简单体现。
<h3 id="四参考">四、参考</h3>
本次实践得益于诸多运维大神对于知识的不吝分享，十分感谢！！！ 
具体参考的博客或指南如下： 
基于 Jenkins 的 CI/CD (一) 强烈推荐阳明老师的博客 
kubernetes实践：安装jenkins slave 
kubernetes Jenkins gitlab搭建CI/CD环境 (二) 
01 [从这里开始]Jenkins CI解决方案 
来源：https://www.cnblogs.com/zhongyuanzhao000/p/11430183.html

頁: [1]

琼殿技术论坛's Archiver

Kubernetes之动态Jenkins slave