Kubernetes Jenkins动态创建Slave

艺术斗争 發表於 2020-3-12 12:35:00

<div class="toc"><div class="toc-container-header">目录</div><ul><li>0、前言</li><li>1、Jenkins部署</li><li>2、配置jenkins动态slave</li><li>3、dubbo服务构建<ul><li>3.1、制作dubbo镜像底包</li><li>3.2、制作slave基础镜像<ul><li>3.2.1、Maven镜像</li><li>3.2.2、Docker镜像</li></ul></li><li>3.3、添加git key</li><li>3.4、创建dubbo流水线</li><li>3.5、执行流水线构建</li></ul></li></ul></div>
<h1 id="0前言">0、前言</h1>
首先，我们考虑个问题，为何需要jenkins slave？其实在生产环境中，如果用单master，除非你单机器的配置特别高并且构建次数不多情况下，可以不考虑使用slave，但是，在构建次数上百次并且jenkins master运行在kubernetes环境中，借助kubernetes的灵活性，强烈推荐使用slave，master负责自动创建Slave Pod，然后将任务推送给Slave Pod，任务执行完毕后，Slave Pod会自动被回收/销毁。
创建slave流程图： 
<img src="https://img2020.cnblogs.com/blog/1679739/202003/1679739-20200312124636614-395855968.png" alt="" loading="lazy">
<h1 id="1jenkins部署">1、Jenkins部署</h1>
jenkins交付进kubernetes
1.准备镜像文件
<pre><code class="language-bash">$ docker pull jenkins/jenkins:2.204.1
$ docker tag a3f949e5ebfd harbor.od.com/infra/jenkins:v2.204.1
$ docker push harbor.od.com/infra/jenkins:v2.204.1
</code></pre>
2.资源配置文件
<ul>
<li>rbac</li>
</ul>
<pre><code class="language-yaml">$ vi /data/k8s-yaml/jenkins_slave/rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: infra
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins
rules:
- apiGroups: ["extensions", "apps"]
resources: ["deployments"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: jenkins
namespace: infra
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
namespace: infra
</code></pre>
Note：Jenkins会自动创建slavepod,所以需要给jenkins绑定Kubernetes的RBAC权限
<ul>
<li>Deployment</li>
</ul>
<pre><code class="language-yaml">$ vi /data/k8s-yaml/jenkins_slave/dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: jenkins
namespace: infra
labels:
name: jenkins
spec:
replicas: 1
selector:
matchLabels:
 name: jenkins
template:
metadata:
 labels:
 app: jenkins
 name: jenkins
spec:
 serviceAccount: jenkins
 volumes:
 - name: data
 nfs:
 server: hdss7-200.host.com
 path: /data/nfs-volume/jenkins_home
 containers:
 - name: jenkins
 image: harbor.od.com/infra/jenkins:v2.204.1
 imagePullPolicy: IfNotPresent
 ports:
 - containerPort: 8080
 name: web
 protocol: TCP
 - containerPort: 50000
 name: agent
 protocol: TCP
 env:
 - name: JAVA_OPTS
 value: "-Xms1G -Xmx1G -XX:PermSize=512m -XX:MaxPermSize=1024m -Duser.timezone=Asia/Shanghai"
 - name: TRY_UPGRADE_IF_NO_MARKER
 value: "true"
 volumeMounts:
 - name: data
 mountPath: /var/jenkins_home
 imagePullSecrets:
 - name: harbor
 securityContext:
 runAsUser: 0
strategy:
type: RollingUpdate
rollingUpdate:
 maxUnavailable: 1
 maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
</code></pre>
<ul>
<li>Service</li>
</ul>
<pre><code class="language-yaml">$ vi /data/k8s-yaml/jenkins_slave/svc.yaml
kind: Service
apiVersion: v1
metadata:
name: jenkins
namespace: infra
spec:
ports:
- name: web
port: 80
targetPort: 8080
protocol: TCP
- name: agent
port: 50000
targetPort: 50000
protocol: TCP
selector:
app: jenkins
</code></pre>
<ul>
<li>Ingress</li>
</ul>
<pre><code class="language-yaml">$ vi /data/k8s-yaml/jenkins_slave/ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: jenkins
namespace: infra
spec:
rules:
- host: jenkins.od.com
http:
 paths:
 - path: /
 backend:
 serviceName: jenkins
 servicePort: 80
</code></pre>
Note：这里的Service对外暴露了80以及50000端口，80作为Jenkins Server Web端口，50000位创建的Jenkins Salve和Master建立通信连接的默认端口，如果该端口不暴露的话，Slave就无法和Master建立连接。
3.应用资源配置清单
<pre><code class="language-bash">$ kubectl apply -f http://k8s-yaml.od.com/jenkins_slave/rbac.yaml
$ kubectl apply -f http://k8s-yaml.od.com/jenkins_slave/dp.yaml
$ kubectl apply -f http://k8s-yaml.od.com/jenkins_slave/svc.yaml
$ kubectl apply -f http://k8s-yaml.od.com/jenkins_slave/ingress.yaml
</code></pre>
<h1 id="2配置jenkins动态slave">2、配置jenkins动态slave</h1>
初始化jenkins后，需要安装kubernetes 插件。
1.安装插件完成之后，点击 Manage Jenkins —> Configure System —> (拖到最下方)Add a new cloud —> 选择 Kubernetes，然后填写 Kubernetes 和 Jenkins 配置信息 
<img src="https://img2020.cnblogs.com/blog/1679739/202003/1679739-20200312123101501-841953565.png" alt="" loading="lazy">
填写kubernetes集群内部访问地址：https://kubernetes.default.svc.cluster.local，点击Test Connection，如果出现 Connection test successful 的提示信息证明 Jenkins 已经可以和 Kubernetes 系统正常通信了，然后下方的 Jenkins URL 地址：http://jenkins.infra.svc.cluster.local:80
<blockquote>
为何我这里使用k8s svc的名称使用方式，为了方便通过DNS记录能够解析成该Service的Cluster IP。
</blockquote>
2.创建Pipeline动态构建测试 
创建一个类型为Pipeline类型Job命名为test-slave，然后在Pipeline脚本填写下面一个简单的测试脚本
<pre><code class="language-shell">def label = "jenkins-slave-${UUID.randomUUID().toString()}"
podTemplate(label: label, cloud: 'kubernetes') {
node(label) {
 stage('Run shell') {
 sh 'sleep 10s'
 sh 'echo hello world.'
 }
}
}
</code></pre>
3.点击构建流水线
可以看到在jenkins的namespaces下自动创建了对应的agent pod 相当于就是一个jenkins 的node 当任务执行完成这个pod会自动退出这个pod默认会去pull一个<code>jenkins/jnlp-slave:x.xx-xx-alpine</code>的镜像
<pre><code class="language-bash"># kubectl get pods -n infra
NAME READY STATUS RESTARTS AGE
jenkins-77b9c47874-qjgfd 1/1 Running 1 13h
jenkins-slave-c07daa7b-31ef-41ea-825e-05c9c721edad-sb7h6-lpgwv 1/1 Running 0 18s
</code></pre>
<h1 id="3dubbo服务构建">3、dubbo服务构建</h1>
我们在构建dubbo服务的时候，需要编译dubbo后制作镜像并推送到harbor，这时候就要用到maven和docker命令，所以需要自己去构建基础镜像，然后在pipeline里调用
<blockquote>
注意：这里并不实现如何将dubbo服务交付到k8s，而是演示实现动态jenkins创建slave去构建dubbo服务
</blockquote>
<h2 id="31制作dubbo镜像底包">3.1、制作dubbo镜像底包</h2>
底包需要有通用性，所有的dubbo微服务制作镜像时都会引用这个底包来制作，该底包主要实现一些功能，例如jmx监控（jmx_javaagent），和dubbo微服务的启动脚本，以及依赖的jdk环境
1.准备镜像（jdk环境）
<pre><code class="language-bash">$ docker pull stanleyws/jre8:8u112
$ docker tag stanleyws/jre8:8u112 harbor.od.com/public/jre:8u112
$ docker push harbor.od.com/public/jre:8u112
</code></pre>
2.自定义Dockerfile
<ul>
<li>/data/dockerfile/jre8/Dockerfile</li>
</ul>
<pre><code class="language-dockerfile">FROM harbor.od.com/public/jre:8u112
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
echo 'Asia/Shanghai' >/etc/timezone
ADD config.yml /opt/prom/config.yml
ADD jmx_javaagent-0.3.1.jar /opt/prom/
WORKDIR /opt/project_dir
ADD entrypoint.sh /entrypoint.sh
CMD ["/entrypoint.sh"]
</code></pre>
<ul>
<li>config.yml（这是jmx_agent读取的配置文件）</li>
</ul>
<pre><code class="language-bash">---
rules:
- pattern: '.*'
</code></pre>
<ul>
<li>jmx_javaagent-0.3.1.jar</li>
</ul>
<pre><code class="language-bash"># 采集jvm监控数据的jar包
$ wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar
</code></pre>
<ul>
<li>entrypoint.sh（不要忘了给该文件执行权限）</li>
</ul>
<pre><code class="language-bash">#!/bin/sh
M_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):${M_PORT:-"12346"}:/opt/prom/config.yml"
C_OPTS=${C_OPTS}
JAR_BALL=${JAR_BALL}
exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL}
</code></pre>
按照上面的java启动选项解释
<pre><code class="language-bash">-agentpath:<pathname>[=<options>] // 加载java代理
-Duser.timezone:<timezone> // 指定时区

C_OPTS=${C_OPTS} // 额外的启动参数，默认为空，可在k8s资源配置清单中添加额外参数
JAR_BALL=${JAR_BALL}// 启动的jar包名字，在k8s资源配置清单中指定
</code></pre>
3.制作dubbo服务docker底包
<pre><code class="language-bash">$ ls -l
total 372
-rw-r--r-- 1 root root 405 Jan 16 15:26 Dockerfile
-rw-r--r-- 1 root root 41 Jan 16 15:28 config.yaml
-rwxr-xr-x 1 root root 234 Jan 16 15:37 entrypoint.sh
-rw-r--r-- 1 root root 367417 May 102018 jmx_prometheus_javaagent-0.3.1.jar

$ docker build . -t harbor.od.com/base/jre8:8u112
$ docker push harbor.od.com/base/jre8:8u112
</code></pre>
<h2 id="32制作slave基础镜像">3.2、制作slave基础镜像</h2>
我们现在有个dubbo项目，需要用到maven构建项目，然后通过Docker打包成镜像并推送到Harbor，所以需要用到两个镜像，Maven以及Docker镜像。
<h3 id="321maven镜像">3.2.1、Maven镜像</h3>
该镜像主要用于构建java应用，这里就选择：maven:v3.3.9-jdk8
准备镜像文件（推送到本地仓库）
<pre><code class="language-bash">$ docker pull maven:3.3.9-jdk-8-alpine
$ docker tag dd9d4e1cd9db harbor.od.com/public/maven:v3.3.9-jdk8
$ docker push harbor.od.com/public/maven:v3.3.9-jdk8
</code></pre>
<h3 id="322docker镜像">3.2.2、Docker镜像</h3>
该镜像主要用于将dubbo项目打包成镜像并推送到harbor，但需要定制化一下镜像，需要将一台已经实现docker login 登录到harbor仓库所生成的配置文件，路径为：<code>/root/.docker/config.json</code>，与原始Docker镜像一起打包生成新的Docker镜像并推送到本地仓库。
1.准备镜像文件
<pre><code class="language-bash">$ docker pull docker:19.03
$ docker tag e036013d6d10 harbor.od.com/public/docker:v19.03
$ docker push harbor.od.com/public/docker:v19.03
</code></pre>
2.Dockerfile
<ul>
<li>vim /data/dockerfile/docker/Dockerfile</li>
</ul>
<pre><code class="language-Dockerfile">FROM harbor.od.com/public/docker:v19.03
USER root
ADD config.json /root/.docker/config.json
</code></pre>
3.将<code>/root/.docker/config.json</code>文件拷贝到Dockerfile目录下
<pre><code class="language-bash">{
"auths": {
"harbor.od.com": {
"auth": "YWRtaW46SGFyYm9yMTIzNDU="
}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/19.03.6 (linux)"
}
}
</code></pre>
<blockquote>
需要通过该文件才能访问到harbor仓库
</blockquote>
4.制作并推送镜像（推送到本地仓库）
<pre><code class="language-bash">$ docker build ./ -t harbor.od.com/public/docker:v19.03
$ docker pushharbor.od.com/public/docker:v19.03
</code></pre>
<h2 id="33添加git-key">3.3、添加git key</h2>
我们这里使用到了Jenkins的Git插件来拉取代码，所以需要先创建一堆密钥，然后将公钥添加到git仓库，再将私钥添加到jenkins凭证，如下： 
<img src="https://img2020.cnblogs.com/blog/1679739/202003/1679739-20200312122120625-1349295856.png" alt="" loading="lazy">
<h2 id="34创建dubbo流水线">3.4、创建dubbo流水线</h2>
1.添加参数化构建
<ul>
<li>
The String Parameter：app_name
Describe：项目名称，例如dubbo-service
</li>
<li>
String Parameter：image_name
Describe：docker镜像名称，格式：<仓库名>/<镜像名> 例如：app/dubbo-demo-service
</li>
<li>
String Parameter：git_repo
Describe：项目所在的git中央仓库的地址：如https://gitee.com/jasonminghao/dubbo-demo-service.git
</li>
<li>
String Parameter：git_ver
Describe：项目所在git中央仓库对应项目的分支或者版本号，例如master分支：*/master，commit ID：903b4e6
</li>
<li>
String Parameter：image_ver
Describe：镜像版本和git_ver一致即可，但是不要有任何的特殊符号
</li>
<li>
String Parameter：add_tag
Describe：docker镜像标签的一部分，日期时间戳，例如：20200121_1734
</li>
<li>
String Parameter：target_dir
Default：./target
Describe：编译项目的目录，产生jar/war包所在的目录
</li>
<li>
String Parameter：mvn_cmd
Default：mvn clean package -Dmaven.test.skip=true
Describe：执行mvn编译所用的命令
</li>
<li>
Choice Parameter：base_image
Choice Value： base/jre8:8u112
Describe：项目使用的docker底包镜像
</li>
</ul>
2.pipeline
<pre><code class="language-bash">podTemplate(cloud:'kubernetes',containers: [
containerTemplate(
 name: 'maven',
 image: 'harbor.od.com/public/maven:v3.3.9-jdk8',
 ttyEnabled: true,
 command: 'cat'),
containerTemplate(
 name: 'docker',
 ttyEnabled: true,
 image: 'harbor.od.com/public/docker:v19.03'),
],
volumes: [
 nfsVolume(mountPath: '/root/.m2', readOnly: false, serverAddress: 'hdss7-200.host.com', serverPath: '/data/nfs-volume/maven_repo/'),
 hostPathVolume(hostPath: '/run/docker.sock', mountPath: '/run/docker.sock')
 ]
){
node(POD_LABEL) {
 stage('Get a Maven project') {
 // 从git仓库拉取代码
 checkout([$class: 'GitSCM', branches: [], browser: [$class: 'GitLab', repoUrl: ''], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: []])

 container('maven') {
 stage('Build a Maven project') {
 // 执行maven构建
 sh "${params.mvn_cmd}"
 }
 }
 }
 stage('Docker build') {
 container('docker') {
 stage('create dir') {
 // /tmp目录创建一个临时用于构建镜像的工作目录，将jar包移动到该目录
 sh "mkdir /tmp/${params.app_name}"
 sh "cd ${params.target_dir} && mkdir /tmp/${params.app_name}/project_dir && mv *.jar /tmp/${params.app_name}/project_dir"
 }
 stage('docker build image') {
 // 动态生成Dockerfile，构建镜像并推送到harbor
 sh "cd /tmp/${params.app_name}/ && ls -lha"
 sh """
 echo "FROM harbor.od.com/${params.base_image}" >/tmp/${params.app_name}/Dockerfile
 echo "ADD ./project_dir /opt/project_dir" >>/tmp/${params.app_name}/Dockerfile
 """
 sh "cd /tmp/${params.app_name}/ && pwd && docker build ./ -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} "
 }
 }
 }
}
}
</code></pre>
<blockquote>
podTemplate只适用于pipeline
</blockquote>
<h2 id="35执行流水线构建">3.5、执行流水线构建</h2>
1.填写对应参数 
<img src="https://img2020.cnblogs.com/blog/1679739/202003/1679739-20200312154051950-377624828.png" alt="" loading="lazy">
2.查看<code>infra</code>名称空间下的Pod
<pre><code class="language-bash"># kubectl get pods -n infra
NAME READY STATUS RESTARTS AGE
apollo-portal-57bc86966d-4tr6w 1/1 Running 8 37h
dubbo-demo-slave-16-trktm-8x2d7-bw5dr 3/3 Running 0 53s
dubbo-monitor-555c94f4b7-85plg 1/1 Running 32 7d14h
jenkins-75fbb46546-f5ltc 1/1 Running 6 18h
</code></pre>
<blockquote>
可以看到jenkins slave会以在jenkins创建的项目名为命名来创建pod
</blockquote>
3.构建结果 
<img src="https://img2020.cnblogs.com/blog/1679739/202003/1679739-20200312154400692-1648808421.png" alt="" loading="lazy"> 
来源：https://www.cnblogs.com/jasonminghao/p/12468407.html

頁: [1]

琼殿技术论坛's Archiver

Kubernetes Jenkins动态创建Slave