黄岩 發表於 2020-12-23 20:30:00

【k8s实战一】Jenkins 部署应用到 Kubernetes

<h1 id="k8s实战一jenkins-部署应用到-kubernetes">【k8s实战一】Jenkins 部署应用到 Kubernetes</h1>
<h2 id="01-本文主旨">01 本文主旨</h2>
<p>目标是演示整个Jenkins从源码构建镜像到部署镜像到Kubernetes集群过程。</p>
<p>为了简化流程与容易重现文中效果，做出如下操作：</p>
<ul>
<li>使用 <code>VBox</code> 创建两台虚拟机</li>
<li>使用 <code>Minikube</code> 初始化简单的k8s集群</li>
<li>使用 <code>GitHub</code> 作为代码仓库，本文内容所有配置文件均会上传至此仓库 https://github.com/hellxz/cicd-demo.git</li>
</ul>
<h2 id="02-cicd流程">02 CI/CD流程</h2>
<p><img src="https://img2020.cnblogs.com/blog/1149398/202012/1149398-20201222111600912-1823409212.png" alt="" loading="lazy"></p>
<p>根据我画的这张图，Jenkins任务整体的流程分为五个步骤：</p>
<ul>
<li>获取源码</li>
<li>构建制品</li>
<li>构建镜像并推送</li>
<li>发起部署请求</li>
<li>k8s调度pod，拉取镜像，部署完成</li>
</ul>
<h2 id="03-准备虚拟机">03 准备虚拟机</h2>
<p>准备两个虚拟机，均安装好docker，参考Ubuntu 16.04及以上 安装/卸载 Docker-CE</p>
<table>
<thead>
<tr>
<th>VM用途</th>
<th>系统版本</th>
<th>IP</th>
</tr>
</thead>
<tbody>
<tr>
<td>Jenkins</td>
<td>Ubuntu 18.04.2 LTS</td>
<td>192.168.87.128</td>
</tr>
<tr>
<td>k8s master</td>
<td>Ubuntu 18.04.2 LTS</td>
<td>192.168.87.129</td>
</tr>
</tbody>
</table>
<p>设置Jenkins机器免密登录k8s master主机</p>
<pre><code class="language-bash">ssh-keygen -t rsa
ssh-copy-id hellxz@192.168.87.129
</code></pre>
<p><img src="https://img2020.cnblogs.com/blog/1149398/202012/1149398-20201222174037249-693714234.png" alt="" loading="lazy"></p>
<p>验证免密效果</p>
<pre><code class="language-bash">ssh hellxz@192.168.87.129
exit
</code></pre>
<h2 id="04-部署registry">04 部署registry</h2>
<p>部署registry之前，可以先将私有仓库配置到 /etc/docker/daemon.json，两台虚拟机均需要配置</p>
<pre><code class="language-bash">sudo mkdir -p /etc/docker
sudo vim /etc/docker/daemon.json
</code></pre>
<p>内容如下：</p>
<pre><code class="language-javascript">{
    "insecure-registries": ["192.168.87.129:5000"]
}
</code></pre>
<p>接着在 192.168.87.129 虚拟机上部署registry，<em>生产环境推荐使用Harbor</em></p>
<pre><code class="language-bash">docker run -d --name registry -p 5000:5000 registry
</code></pre>
<p><img src="https://img2020.cnblogs.com/blog/1149398/202012/1149398-20201222174626008-1105184244.png" alt="" loading="lazy"></p>
<h2 id="05-配置maven">05 配置maven</h2>
<p>在jenkins虚拟机安装maven</p>
<pre><code class="language-bash">wget https://mirrors.aliyun.com/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
tar zxvf apache-maven-3.6.3-bin.tar.gz
mv apache-maven-3.6.3 maven-3.6.3
</code></pre>
<p>修改 <code>/etc/profile</code>，添加 <code>M2_HOME</code> 环境变量映射到 maven 目录，重写 <code>PATH</code> 添加 maven 的 bin 目录，如图31、32行</p>
<p><img src="https://img2020.cnblogs.com/blog/1149398/202012/1149398-20201223154657800-187420189.png" alt="" loading="lazy"></p>
<h2 id="06-tomcat部署jenkins">06 Tomcat部署Jenkins</h2>
<p>在jenkins虚拟机 192.168.87.128上执行命令下载 tomcat 与 jenkins.war</p>
<pre><code class="language-bash">#下载tomcat
wget https://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-9/v9.0.41/bin/apache-tomcat-9.0.41.tar.gz
#解压并重命名
tar -zxf apache-tomcat-9.0.41.tar.gz
mv apache-tomcat-9.0.41 /home/hellxz/jenkins
#进入jenkins安装目录
cd ~/jenkins/webapps
#删除webapps下示例项目
rm -rf *
#下载最新版jenkins.war
wget https://mirrors.ustc.edu.cn/jenkins/war/latest/jenkins.war
#安装openjdk、git
sudo apt-update
sudo apt-get install -y default-jdk default-jdk-headless git
</code></pre>
<p>设置Jenkins家目录，编辑 /etc/profile，添加 <code>JENKINS_HOME</code> 环境变量</p>
<p><img src="https://img2020.cnblogs.com/blog/1149398/202012/1149398-20201222181748983-1643546983.png" alt="" loading="lazy"></p>
<p>刷新配置</p>
<pre><code class="language-bash">source /etc/profile
</code></pre>
<p>修改tomcat下 <code>conf/context.xml</code>，加大静态资源缓存，提前解决启动 Jenkins 时的报错</p>
<pre><code class="language-xml">&lt;Resources cachingAllowed="true" cacheMaxSize="9999999" /&gt;
</code></pre>
<p><img src="https://img2020.cnblogs.com/blog/1149398/202012/1149398-20201222190236412-1972452625.png" alt="" loading="lazy"></p>
<p>启动Jenkins</p>
<pre><code class="language-bash">cd ~/jenkins/bin
./startup.sh
</code></pre>
<p><img src="https://img2020.cnblogs.com/blog/1149398/202012/1149398-20201222182716806-1477198651.png" alt="" loading="lazy"></p>
<p>访问 http://192.168.87.129:8080，安装推荐插件，安装加速可参考 Jenkins安装插件提速</p>
<h2 id="07-部署kubernetes集群">07 部署Kubernetes集群</h2>
<p>部署一个可以通过 <code>kubectl</code> 命令部署应用的k8s环境即可，部署k8s环境不是本文重点，以下为参考</p>
<details>
    <summary>使用kubeadm部署Kubernetes集群，点击展开查看</summary>
    参考我之前的博客吧，差异应该不大<br>
    【k8s学习笔记】使用 kubeadm 部署 v1.18.5 版本 Kubernetes集群
</details>
<details>
        <summary>使用Minikube部署单节点测试k8s环境，点击展开查看</summary>
        <pre><blockcode>
#添加阿里apt kubernetes源
sudo apt-get update &amp;&amp; sudo apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -
sudo tee /etc/apt/sources.list.d/kubernetes.list &lt;&lt;-'EOF'
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
sudo apt-get update
#安装kubectl
sudo apt-get install -y kubectl
#下载minikube二进制文件
curl -Lo minikube https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v1.16.0/minikube-linux-amd64 &amp;&amp; chmod +x minikube &amp;&amp; sudo mv minikube /usr/local/bin/
#部署k8s环境，默认使用docker驱动。读者也可使用vbox
minikube start -–insecure-registry=192.168.87.129:5000
</blockcode></pre>
<img src="https://img2020.cnblogs.com/blog/1149398/202012/1149398-20201223171022823-1706148598.png">
</details>
<blockquote>
<p>本文使用Minikube做测试，坑比较多，想趟坑可以先参考下 https://www.kubernetes.org.cn/6388.html</p>
<details>
    <summary>关于Minikube的吐槽</summary>
<p>
写文时，用的Win10创建Linux无桌面虚拟机，在虚拟机基础上装的minikube，minikube默认使用Docker在虚拟机中又创建了个Docker容器，里边才是k8s环境，所以想在虚拟机外访问套了两层的内网应用服务，可能还需要在虚拟机中加Nginx，通过它转发请求到虚拟机中的内网地址……虽然疯狂套娃挺坑的，但为了演示下效果还是可以的。
PS：在Linux/MacOS环境直接装Minikube，是可以通过浏览器直接看的。由于未直接在Win环境使用Docker Desktop，所以未确定Win下是否可用。</p>
</details>
</blockquote>
<h2 id="08-准备demo">08 准备Demo</h2>
<p>Demo分五部分：测试代码、Jenkins构建脚本、docker镜像构建脚本、k8s部署配置yaml、部署脚本，均处在同一仓库中</p>
<p>仓库地址：https://github.com/hellxz/cicd-demo.git</p>
<p>目录结构：</p>
<p><img src="https://img2020.cnblogs.com/blog/1149398/202012/1149398-20201223192624953-1729524056.png" alt="" loading="lazy"></p>
<h3 id="demo">demo</h3>
<p>使用start.spring.io生成的只添加spring-boot-starter-web依赖，在启动类上加了个 <code>/test</code> 的接口</p>
<p><img src="https://img2020.cnblogs.com/blog/1149398/202012/1149398-20201223192735940-1042081634.png" alt="" loading="lazy"></p>
<h3 id="dockerfile">Dockerfile</h3>
<pre><code class="language-bash">FROM openjdk:8-jdk
#设置工作目录
WORKDIR /root
#复制制品jar到/root/app.jar位置
ADD target/*.jar app.jar
#JVM参数，给后期调优使用
ENV JVM_OPTS=""
#启动服务
CMD java ${JVM_OPTS} \
    -Djava.security.egd=file:/dev/./urandom \
    -jar app.jar
</code></pre>
<h3 id="shell">shell</h3>
<p><code>artifact2image.sh</code> 制品转镜像Shell脚本，本文为了简单未安装Jenkins关于docker的插件</p>
<pre><code class="language-bash">#!/bin/bash
# -- 构建镜像并推送私有仓库 --
set -eu #如有报错或取不到变量情况停止执行

#声名常量
IMG_REGISTRY="192.168.87.129:5000" #镜像仓库
IMG_NAME="cicd-demo"
IMG_TAG=`date "+%Y%M%d_%H%M"`      #镜像标签，如 20201223_1351
IMG_FULL_NAME="${IMG_REGISTRY}/${IMG_NAME}:${IMG_TAG}" #镜像上传与拉取的名称

#构建镜像
docker build -t ${IMG_FULL_NAME} .

#推送镜像
docker push ${IMG_FULL_NAME}

#删除本地镜像
docker rmi ${IMG_FULL_NAME}

#修改deploy.yaml的镜像标签
sed -i "s#{{IMAGE_NAME}}#${IMG_FULL_NAME}#g" deploy.yaml
</code></pre>
<p><code>deploy2k8s.sh</code> 部署镜像到k8s环境脚本</p>
<pre><code class="language-bash">#!/bin/bash
# -- 部署image到k8s --
# !注意：需要提前做ssh免密登录
set -eu

#定义常量
PROJECT_NAME="cicd-demo"
UPLOAD_DIR="/home/hellxz/apps/${PROJECT_NAME}"
FILE_NAME="${UPLOAD_DIR}/deploy.yaml"
SSH_USER="hellxz"
SSH_IP="192.168.87.129"

#首先删除待上传目录的同名文件
ssh ${SSH_USER}@${SSH_IP} "rm -rf ${FILE_NAME}"

#确保部署文件目录存在
ssh ${SSH_USER}@${SSH_IP} "mkdir -p ${UPLOAD_DIR}"

#远程复制部署文件
scp -r deploy.yaml ${SSH_USER}@${SSH_IP}:${FILE_NAME}

#远程执行部署命令
ssh ${SSH_USER}@${SSH_IP} "kubectl apply -f ${FILE_NAME}"
</code></pre>
<h3 id="jenkinsfile">Jenkinsfile</h3>
<pre><code class="language-groovy">pipeline{
    agent any
    stages {
      //由于源码和Jenkinsfile处于同一仓库，在Jenkins项目执行时，会先将Jenkinsfile所在的仓库克隆下来，为了简单，这里就不重复添加拉取源码的操作了
      //其它情况，如使用多个仓库一些构建的，这种就需要额外添加拉取代码的stage了。
      stage('Build Artifact') {
            steps{
                sh label:'maven building', script: 'mvn clean package -DskipTests'
            }
      }
      stage('Build Image'){
            steps{
                sh label:'image building', script: '/bin/bash artifact2image.sh'
            }
      }
      stage('Deploy k8s'){
            steps{
                sh label:'deploy image to k8s', script: '/bin/bash deploy2k8s.sh'
            }
      }
    }
    post {
      success{
            //成功清理工作空间，失败保留现场
            cleanWs()
      }
    }
}

</code></pre>
<h3 id="deployyaml">deploy.yaml</h3>
<pre><code class="language-yaml">apiVersion: apps/v1   #api版本
kind: Deployment      #对象类型
metadata:             #元信息标识
name: cicd-demo
namespace: default
labels:             #自定义Deployment绑定的标签
    app: cicd-demo
spec:               #期待运行状态
replicas: 1         #部署实例数
selector:
    matchLabels:
      app: cicd-demo
template:         #pod
    metadata:
      labels:         #与spec.selector.matchLabels相同
      app: cicd-demo
    spec:             #pod容器期望的状态
      containers:
      - name: cicd-demo
          image: {{IMAGE_NAME}}
          imagePullPolicy: IfNotPresent
          env:
            - name: JVM_OPTS
            value: "-Xms128m -Xmx256m"
          ports:
            - containerPort: 8080

---
apiVersion: v1
kind: Service
metadata:
name: cicd-demo-svc
namespace: default
labels:
    service: cicd-demo-svc
spec:
selector:            #匹配pod标签
    app: cicd-demo
type: NodePort   #svc类型，ExternalName, ClusterIP, NodePort, and LoadBalancer
ports:
    - name: cicd-demo-port
      protocol: TCP
      port: 8080
      nodePort: 30000

</code></pre>
<h2 id="09-创建jenkins流水线项目">09 创建Jenkins流水线项目</h2>
<p>访问 http://192.168.87.128:8080/jenkins 并登录，以下创建新流水线demo构建项目</p>
<p><img src="https://img2020.cnblogs.com/blog/1149398/202012/1149398-20201223194342876-1007447970.gif" alt="" loading="lazy"></p>
<h2 id="部署与验证">部署与验证</h2>
<p>执行构建demo程序</p>
<p><img src="https://img2020.cnblogs.com/blog/1149398/202012/1149398-20201223194948410-153070889.png" alt="" loading="lazy"></p>
<p><img src="https://img2020.cnblogs.com/blog/1149398/202012/1149398-20201223195033551-437763942.png" alt="" loading="lazy"></p>
<p>日志输出：</p>
<pre><code class="language-bash">Started by user admin
Obtained Jenkinsfile from git https://github.com/hellxz/cicd-demo.git
Running in Durability level: MAX_SURVIVABILITY
Start of Pipeline
node
Running on Jenkins in /home/hellxz/jenkins/home/workspace/demo
{
stage
{ (Declarative: Checkout SCM)
checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/hellxz/cicd-demo.git
&gt; git init /home/hellxz/jenkins/home/workspace/demo # timeout=10
Fetching upstream changes from https://github.com/hellxz/cicd-demo.git
&gt; git --version # timeout=10
&gt; git --version # 'git version 2.17.1'
&gt; git fetch --tags --progress -- https://github.com/hellxz/cicd-demo.git +refs/heads/*:refs/remotes/origin/* # timeout=10
&gt; git config remote.origin.url https://github.com/hellxz/cicd-demo.git # timeout=10
&gt; git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
&gt; git rev-parse refs/remotes/origin/master^{commit} # timeout=10
Checking out Revision 1fc086590e32c001c8b65a4c575f4aa1b5b20413 (refs/remotes/origin/master)
&gt; git config core.sparsecheckout # timeout=10
&gt; git checkout -f 1fc086590e32c001c8b65a4c575f4aa1b5b20413 # timeout=10
Commit message: "update"
&gt; git rev-list --no-walk 1fc086590e32c001c8b65a4c575f4aa1b5b20413 # timeout=10
}
// stage
withEnv
{
stage
{ (Build Artifact)
sh (maven building)
+ mvn clean package -DskipTests
NOTE: Picked up JDK_JAVA_OPTIONS:--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
Scanning for projects...


----------------------&lt; online.hellxz:cicd-demo &gt;-----------------------
Building cicd-demo 0.0.1
--------------------------------[ jar ]---------------------------------

--- maven-clean-plugin:3.1.0:clean (default-clean) @ cicd-demo ---

--- maven-resources-plugin:3.2.0:resources (default-resources) @ cicd-demo ---
Using 'UTF-8' encoding to copy filtered resources.
Using 'UTF-8' encoding to copy filtered properties files.
Copying 1 resource
Copying 0 resource
The encoding used to copy filtered properties files have not been set. This means that the same encoding will be used to copy filtered properties files as when copying other filtered resources. This might not be what you want! Run your build with --debug to see which files might be affected. Read more at https://maven.apache.org/plugins/maven-resources-plugin/examples/filtering-properties-files.html

--- maven-compiler-plugin:3.8.1:compile (default-compile) @ cicd-demo ---

Changes detected - recompiling the module!
Compiling 1 source file to /home/hellxz/jenkins/home/workspace/demo/target/classes


--- maven-resources-plugin:3.2.0:testResources (default-testResources) @ cicd-demo ---
Using 'UTF-8' encoding to copy filtered resources.
Using 'UTF-8' encoding to copy filtered properties files.
skip non existing resourceDirectory /home/hellxz/jenkins/home/workspace/demo/src/test/resources

--- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ cicd-demo ---
No sources to compile

--- maven-surefire-plugin:2.22.2:test (default-test) @ cicd-demo ---
Tests are skipped.

--- maven-jar-plugin:3.2.0:jar (default-jar) @ cicd-demo ---
Building jar: /home/hellxz/jenkins/home/workspace/demo/target/cicd-demo-0.0.1.jar

--- spring-boot-maven-plugin:2.4.1:repackage (repackage) @ cicd-demo ---
Replacing main artifact with repackaged archive
------------------------------------------------------------------------
BUILD SUCCESS
------------------------------------------------------------------------
Total time:3.248 s
Finished at: 2020-12-23T11:51:20Z
------------------------------------------------------------------------
}
// stage
stage
{ (Build Image)

sh (image building)
+ /bin/bash artifact2image.sh
Sending build context to Docker daemon17.26MB

Step 1/5 : FROM openjdk:8-jdk
---&gt; 89f100fa8f9f
Step 2/5 : WORKDIR /root
---&gt; Using cache
---&gt; 17670bfa01f1
Step 3/5 : ADD target/*.jar app.jar

---&gt; b7e4292e4f80
Step 4/5 : ENV JVM_OPTS=""
---&gt; Running in 0be8c16c9c8a
Removing intermediate container 0be8c16c9c8a
---&gt; 3ef66b100082
Step 5/5 : CMD java -Djava.security.egd=file:/dev/./urandom   ${JVM_OPTS}   -jar app.jar
---&gt; Running in c8209a642a2d
Removing intermediate container c8209a642a2d
---&gt; bc7859f9b4fe
Successfully built bc7859f9b4fe
Successfully tagged 192.168.87.129:5000/cicd-demo:20205123_1151

The push refers to repository
44cbb90cc2c3: Preparing
f85e383859a1: Preparing
ffb4778f8a52: Preparing
e528f2c31deb: Preparing
c5f4367d4a59: Preparing
ceecb62b2fcc: Preparing
193bc1d68b80: Preparing
f0e10b20de19: Preparing
ceecb62b2fcc: Waiting
193bc1d68b80: Waiting
f0e10b20de19: Waiting
ffb4778f8a52: Layer already exists
e528f2c31deb: Layer already exists
f85e383859a1: Layer already exists
193bc1d68b80: Layer already exists
c5f4367d4a59: Layer already exists
ceecb62b2fcc: Layer already exists
f0e10b20de19: Layer already exists
44cbb90cc2c3: Pushed

20205123_1151: digest: sha256:24976370741b8e20a7e8c7d18e13f4b72bb492f9c77908927d3b17b8e1ce75d4 size: 2006
Untagged: 192.168.87.129:5000/cicd-demo:20205123_1151
Untagged: 192.168.87.129:5000/cicd-demo@sha256:24976370741b8e20a7e8c7d18e13f4b72bb492f9c77908927d3b17b8e1ce75d4
Deleted: sha256:bc7859f9b4fefc5fdcb39bab4c77dff94980d70fcd21b0b3755f317f517e4291
Deleted: sha256:3ef66b100082a9d6949802c53fe57442d0e156216f0ba7a544b4e44d5a80e7a4
Deleted: sha256:b7e4292e4f807a13f800b5acc8adb0a84926e5ce5c4cc850a2769503558efcb9
Deleted: sha256:2afe950a3c18304e26f7d6b3f3bc48d9de631eb8827203bfa273d361647d0951
}
// stage
stage
{ (Deploy k8s)
sh (deploy image to k8s)
+ /bin/bash deploy2k8s.sh

deployment.apps/cicd-demo created
service/cicd-demo-svc created
}
// stage
stage
{ (Declarative: Post Actions)

cleanWs
Deleting project workspace...
Deferred wipeout is used...
done
}
// stage
}
// withEnv
}
// node
End of Pipeline
Finished: SUCCESS

</code></pre>
<p>可以看到以下输出，说明程序已部署到k8s集群中了</p>
<pre><code class="language-bash">deployment.apps/cicd-demo created
service/cicd-demo-svc created
</code></pre>
<p>ssh登录k8s的机器验证</p>
<p><img src="https://img2020.cnblogs.com/blog/1149398/202012/1149398-20201223195926453-1496495818.png" alt="" loading="lazy"></p>
<h2 id="总结">总结</h2>
<p>简单总结下，Jenkins与Kubernetes的集成，更多地是Jenkins如何控制Kubernetes部署的流程，文中例子是通过 <code>构建并上传镜像</code> 和 <code>ssh发送配置和命令</code> 实现的。</p>
<p>本文是参考慕课网上的教程，快速扫了下视频实践所得，是了解其思路后自行实现的。脚本还有一点不完美，比如替换部署yaml内容是一次性的，这里只是演示大家知道整体流程就好。</p>
<p>实际操作上才发现，脑子说会了，然后双手却没有跟上，还是要动手啊！</p>
<p>操作了一天半左右，最开始还想着用vagrant初始化VM，由于工作上有个虚拟机是VBox5的，升级新版不能用。搞了半天放弃了Vagrant（VBox6.1升级了命令参数不兼容旧版，导致Vagrant为兼容它把几乎所有版本Vagrant都改了……）。</p>
<p>文中难免因为安装部分的描述而过于冗长，感谢大家能看到这里，最后再次感谢慕课网的鹿哥</p>
<h2 id="参考">参考</h2>
<p>https://www.imooc.com/learn/1112<br>
https://cloud.tencent.com/developer/article/1394657<br>
https://www.jianshu.com/p/3f13850d66ea<br>
https://www.cnblogs.com/hellxz/p/use-kubeadm-init-kubernetes-cluster.html<br>
https://www.cnblogs.com/hellxz/p/how-kubernetes-deploy-application.html<br>
https://developer.aliyun.com/article/221687?spm=a2c6h.12873639.0.0.755c6235EZuEht<br>
https://minikube.sigs.k8s.io/docs/start<br>
https://www.cnblogs.com/hellxz/p/install_jenkins.html<br>
https://www.cnblogs.com/hellxz/p/jenkins_install_plugins_faster.html</p><br><br>
来源：https://www.cnblogs.com/hellxz/p/easyway_jenkins_integration_k8s.html
頁: [1]
查看完整版本: 【k8s实战一】Jenkins 部署应用到 Kubernetes