kubernetes监控终极方案-kube-promethues

慈葡發表於 2019-9-7 15:15:00

<h2>kube-promethues简介</h2>
前面我们学习了Heapster+cAdvisor方式监控，这是Prometheus Operator出现之前的k8s监控方案。后来出现了Prometheus Operator，但是目前Prometheus Operator已经不包含完整功能，完整的解决方案已经变为kube-prometheus。项目地址为：https://github.com/coreos/kube-prometheus
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907092904269-1984828303.png" alt="">
这个仓库包括：kubernetes清单、granfana dashboard以及promethues rules。同时还包括容易上手的安装脚本。
组件包括：
<ul>
<li>The Prometheus Operator</li>
<li>高可用Prometheus</li>
<li>高可用Alertmanager</li>
<li>Prometheus node-exporter</li>
<li>Prometheus Adapter for Kubernetes Metrics APIs</li>
<li>kube-state-metrics</li>
<li>Grafana</li>
</ul>
<h2>kube-promethues架构</h2>
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907093555092-1273384033.png" alt="">
上图是<code>Prometheus-Operator</code>官方提供的架构图，其中<code>Operator</code>是最核心的部分，作为一个控制器，他会去创建<code>Prometheus</code>、<code>ServiceMonitor</code>、<code>AlertManager</code>以及<code>PrometheusRule</code>4个<code>CRD</code>资源对象，然后会一直监控并维持这4个资源对象的状态。
其中创建的<code>prometheus</code>这种资源对象就是作为<code>Prometheus Server</code>存在，而<code>ServiceMonitor</code>就是<code>exporter</code>的各种抽象，<code>exporter</code>前面我们已经学习了，是用来提供专门提供<code>metrics</code>数据接口的工具，<code>Prometheus</code>就是通过<code>ServiceMonitor</code>提供的<code>metrics</code>数据接口去 pull 数据的，当然<code>alertmanager</code>这种资源对象就是对应的<code>AlertManager</code>的抽象，而<code>PrometheusRule</code>是用来被<code>Prometheus</code>实例使用的报警规则文件。
这样我们要在集群中监控什么数据，就变成了直接去操作 Kubernetes 集群的资源对象了，是不是方便很多了。上图中的 Service 和 ServiceMonitor 都是 Kubernetes 的资源，一个 ServiceMonitor 可以通过 labelSelector 的方式去匹配一类 Service，Prometheus 也可以通过 labelSelector 去匹配多个ServiceMonitor。
<h2>kube-promethues部署</h2>
下载安装源码
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">git clone https://github.com/coreos/kube-prometheus.git
</pre>
</div>
　　安装文件都在kube-prometheus/manifests/ 目录下。
官方把所有文件都放在一起,这里我复制了然后分类下
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">mkdir prometheus
cp kube-prometheus/manifests/* prometheus/
cd prometheus/
mkdir -p operator node-exporter alertmanager grafana kube-state-metrics prometheus serviceMonitor adapter
mv *-serviceMonitor* serviceMonitor/
mv 0prometheus-operator* operator/
mv grafana-* grafana/
mv kube-state-metrics-* kube-state-metrics/
mv alertmanager-* alertmanager/
mv node-exporter-* node-exporter/
mv prometheus-adapter* adapter/
mv prometheus-* prometheus/
</pre>
</div>
　　注意：新版本的默认label变了，需要修改选择器为<code>beta.kubernetes.io/os，不然安装的时候会卡住。</code>
修改选择器
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">sed -ri '/linux/s#kubernetes.io#beta.&#' \
alertmanager/alertmanager-alertmanager.yaml \
prometheus/prometheus-prometheus.yaml \
node-exporter/node-exporter-daemonset.yaml \
kube-state-metrics/kube-state-metrics-deployment.yaml
</pre>
</div>
 
镜像使用dockerhub上的
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">sed -ri '/quay.io/s#quay.io/prometheus#prom#' \
alertmanager/alertmanager-alertmanager.yaml \
prometheus/prometheus-prometheus.yaml \
node-exporter/node-exporter-daemonset.yaml
</pre>
</div>
　　
使用能拉取到的谷歌镜像
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">find -type f -exec sed -ri 's#k8s.gcr.io#gcr.azk8s.cn/google_containers#' {} \;
</pre>
</div>
　　
当前文件目录：
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907151056439-1714099461.png" alt="">
 
 
1、生成namespace
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">kubectl apply -f .
</pre>
</div>
　
2、安装operater
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">kubectl apply -f operator/
</pre>
</div>
　　
3、依次安装其他组件
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">kubectl apply -f adapter/
kubectl apply -f alertmanager/
kubectl apply -f node-exporter/
kubectl apply -f kube-state-metrics/
kubectl apply -f grafana/
kubectl apply -f prometheus/
kubectl apply -f serviceMonitor/
</pre>
</div>
　　
4、查看整体状态
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">kubectl -n monitoring get all
</pre>
</div>
　　
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;"># kubectl -n monitoring get all
NAME READY STATUS RESTARTS AGE
pod/alertmanager-main-0 2/2 Running 0 27h
pod/alertmanager-main-1 2/2 Running 0 27h
pod/alertmanager-main-2 2/2 Running 0 27h
pod/grafana-7b86fd9ffd-sslwf 1/1 Running 0 27h
pod/kube-state-metrics-688965c565-knjbz 4/4 Running 0 27h
pod/node-exporter-4vtgl 2/2 Running 0 27h
pod/node-exporter-5bnfw 2/2 Running 0 27h
pod/node-exporter-9nnsp 2/2 Running 0 27h
pod/node-exporter-fd8ng 2/2 Running 0 27h
pod/node-exporter-nh5q9 2/2 Running 0 27h
pod/node-exporter-z69fb 2/2 Running 0 27h
pod/prometheus-adapter-66fc7797fd-qpt4d 1/1 Running 0 27h
pod/prometheus-k8s-0 3/3 Running 1 27h
pod/prometheus-k8s-1 3/3 Running 1 27h
pod/prometheus-operator-78678c7494-6954s 1/1 Running 0 27h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager-main ClusterIP 10.254.160.2 <none> 9093/TCP 27h
service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 27h
service/grafana LoadBalancer 10.254.2.98 120.92.212.201 3000:31423/TCP 27h
service/kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 27h
service/node-exporter ClusterIP None <none> 9100/TCP 27h
service/prometheus-adapter ClusterIP 10.254.225.221 <none> 443/TCP 27h
service/prometheus-k8s LoadBalancer 10.254.23.154 120.92.92.56 9090:32361/TCP 27h
service/prometheus-operated ClusterIP None <none> 9090/TCP 27h
service/prometheus-operator ClusterIP None <none> 8080/TCP 27h

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/node-exporter 6 6 6 6 6 beta.kubernetes.io/os=linux 27h

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/grafana 1/1 1 1 27h
deployment.apps/kube-state-metrics 1/1 1 1 27h
deployment.apps/prometheus-adapter 1/1 1 1 27h
deployment.apps/prometheus-operator 1/1 1 1 27h

NAME DESIRED CURRENT READY AGE
replicaset.apps/grafana-7b86fd9ffd 1 1 1 27h
replicaset.apps/kube-state-metrics-688965c565 1 1 1 27h
replicaset.apps/kube-state-metrics-758f8b9855 0 0 0 27h
replicaset.apps/prometheus-adapter-66fc7797fd 1 1 1 27h
replicaset.apps/prometheus-operator-78678c7494 1 1 1 27h

NAME READY AGE
statefulset.apps/alertmanager-main 3/3 27h
statefulset.apps/prometheus-k8s 2/2 27h
</pre>
</div>
　　
 
<h2>kube-promethues配置</h2>
1、kube-controller-manager 和 kube-scheduler组件配置
我们可以看到大部分的配置都是正常的，只有两三个没有管理到对应的监控目标，比如 kube-controller-manager 和 kube-scheduler 这两个系统组件，这就和 ServiceMonitor 的定义有关系了，我们先来查看下 kube-scheduler 组件对应的 ServiceMonitor 资源的定义：(prometheus-serviceMonitorKubeScheduler.yaml)
 
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: kube-scheduler
name: kube-scheduler
namespace: monitoring
spec:
endpoints:
- interval: 30s # 每30s获取一次信息
port: http-metrics# 对应service的端口名
jobLabel: k8s-app
namespaceSelector: # 表示去匹配某一命名空间中的service，如果想从所有的namespace中匹配用any: true
matchNames:
- kube-system
selector:# 匹配的 Service 的labels，如果使用mathLabels，则下面的所有标签都匹配时才会匹配该service，如果使用matchExpressions，则至少匹配一个标签的service都会被选择
matchLabels:
 k8s-app: kube-scheduler
</pre>
</div>
　　上面是一个典型的 ServiceMonitor 资源文件的声明方式，上面我们通过<code>selector.matchLabels</code>在 kube-system 这个命名空间下面匹配具有<code>k8s-app=kube-scheduler</code>这样的 Service，但是我们系统中根本就没有对应的 Service，所以我们需要手动创建一个 Service：（prometheus-kubeSchedulerService.yaml）
cat prometheus-kubeSchedulerService.yaml
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">apiVersion: v1
kind: Service
metadata:
namespace: kube-system
name: kube-scheduler
labels:
k8s-app: kube-scheduler
spec:
type: ClusterIP
clusterIP: None
ports:
- name: port
port: 10251
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
labels:
k8s-app: kube-scheduler
name: kube-scheduler
namespace: kube-system
subsets:
- addresses:
- ip: 10.0.0.5
- ip: 10.0.0.15
- ip: 10.0.0.20
ports:
- name: http-metrics
port: 10251
protocol: TCP
</pre>
</div>
　　
　　同理prometheus-kubeControllerManagerService.yaml也需要修改一下：
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">apiVersion: v1
kind: Service
metadata:
namespace: kube-system
name: kube-controller-manager
labels:
k8s-app: kube-controller-manager
spec:
selector:
component: kube-controller-manager
type: ClusterIP
clusterIP: None
ports:
- name: http-metrics
port: 10252
targetPort: 10252
protocol: TCP

---
apiVersion: v1
kind: Endpoints
metadata:
labels:
k8s-app: kube-controller-manager
name: kube-controller-manager
namespace: kube-system
subsets:
- addresses:
- ip: 10.0.0.5
- ip: 10.0.0.15
- ip: 10.0.0.20
ports:
- name: http-metrics
port: 10252
protocol: TCP
</pre>
</div>
　　
 2、coredns配置
我在测试环境和线上环境都遇到了coredns无法发现的问题。
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907161740355-1677813924.png" alt="">
 
 
 出现这个问题的原因在于/data/monitor/kube-prometheus/manifests/prometheus-serviceMonitorCoreDNS.yaml这个文件：
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907161932547-1076268584.png" alt="">
 
 
 上面的matchLabels为kube-dns，而实际上不是这个label：
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">pod/coredns-58d6869b44-ddczz 1/1 Running 0 4d4h 10.8.5.2 10.0.0.5 <none> <none>
pod/coredns-58d6869b44-zrkx4 1/1 Running 0 4d4h 10.8.4.2 10.0.0.15 <none> <none>

service/coredns ClusterIP 10.254.0.10 <none> 53/UDP,53/TCP,9153/TCP 4d4h k8s-app=coredns
</pre>
</div>
　　实际上是coredns，所以需要修改。
同时，rules里面也需要修改。
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907162956379-1455895187.png" alt="">
 
 
 <img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907163029358-1478768185.png" alt="">
 
 
 修改完成后：
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907163105890-1937097340.png" alt="">
 
 
 
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907163126378-320218009.png" alt="">
 
 
 恢复正常。
 
3、自定义监控项
除了 Kubernetes 集群中的一些资源对象、节点以及组件需要监控，有的时候我们可能还需要根据实际的业务需求去添加自定义的监控项，添加一个自定义监控的步骤也是非常简单的。
<ul>
<li>第一步建立一个 ServiceMonitor 对象，用于 Prometheus 添加监控项</li>
<li>第二步为 ServiceMonitor 对象关联 metrics 数据接口的一个 Service 对象</li>
<li>第三步确保 Service 对象可以正确获取到 metrics 数据</li>
</ul>
比如，在我这套环境中，业务那边在测试微服务，启动了2个pod。
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907163519360-903106767.png" alt="">
 同时他开放了7000端口作为数据接口：
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907163445393-166643844.png" alt="">
 
新建serviceMonitor文件:
cat prometheus-serviceMonitorJx3recipe.yaml
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: jx3recipe
namespace: monitoring
labels:
k8s-app: jx3recipe
spec:
jobLabel: k8s-app
endpoints:
- port: port
interval: 30s
scheme: http
selector:
matchLabels:
 run: jx3recipe
namespaceSelector:
matchNames:
- default
</pre>
</div>
　　
新建service文件
cat prometheus-jx3recipeService.yaml
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">apiVersion: v1
kind: Service
metadata:
name: jx3recipe
namespace: default
labels:
run: jx3recipe
spec:
type: ClusterIP
clusterIP: None
ports:
- name: port
port: 7000
protocol: TCP

---
apiVersion: v1
kind: Endpoints
metadata:
name: jx3recipe
namespace: default
labels:
run: jx3recipe
subsets:
- addresses:
- ip: 10.8.0.19
nodeName: jx3recipe-01
- ip: 10.8.2.17
nodeName: jx3recipe-02
ports:
- name: port
port: 7000
protocol: TCP
</pre>
</div>
　　这里有个问题，我把EP写死了，可能容器重启后，IP地址改变的话，就不行了，所以这里有点问题。
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:true;">kubectl apply -fprometheus-jx3recipeService.yaml
kubectl apply -fprometheus-serviceMonitorJx3recipe.yaml
</pre>
</div>
　　
在promethues的targets里面可以看到效果：
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907163946095-894043990.png" alt="">
 
 
随便挑个item查看数据：
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907164045516-1947960201.png" alt="">
 
 
 一切OK，后续我们就可以使用grafana定制dashboard了。
<h2>kube-promethues对外暴露</h2>
上面环境搭建完成后，服务都是对内的，我们需要对外进行暴露，由于环境是搭建在金山云上，所以直接使用公有云的LB即可。
grafana暴露
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907165400353-1298079404.png" alt="">
 
 只需要把service的type改为LB。
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907165502359-1040100564.png" alt="">
 
 直接通过外网IP访问即可。http://120.92.*.*:3000访问grafana。
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907165640164-1322300985.png" alt="">
 
 
进入后可以看到已经自带了很多dashboard，比较丰富，包括cluster、node、pod以及k8s的各个组件的监控数据。
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907165758359-1008182064.png" alt="">
 
 
　
Node
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907165826744-1724181362.png" alt="">
 
 
 promethues本身
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907165853356-431740999.png" alt="">
 
 
当然，我们还可以从grafana导入一些更加炫酷的dashboard，便于我们实时掌握资源使用情况。
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907170710367-338912633.png" alt="">
 
 相关文件已经上传到了github上，可以直接导入使用：
https://github.com/loveqx/k8s-study
<img src="https://img2018.cnblogs.com/blog/462684/201909/462684-20190907170755709-1141894134.png" alt="">
 
 
<pre></pre>
  
来源：https://www.cnblogs.com/skyflask/p/11480988.html

頁: [1]

琼殿技术论坛's Archiver

kubernetes监控终极方案-kube-promethues