4.prometheus监控-监控linux服务器 - Powered by Discuz! Archiver

有明發表於 2024-4-24 16:07:00

4.prometheus监控--监控linux服务器

<h3>一、监控linux服务器</h3>
<h4>1.1 二进制安装</h4>
<div class="cnblogs_code">
<pre># 客户端操作 wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz

tar xvf node_exporter-1.7.0.linux-amd64.tar.gz

ls -l

mv node_exporter-1.7.0.linux-amd64/* /opt/prometheus/node_exporter useradd -M -s /usr/sbin/nologin prometheus chown prometheus:prometheus -R /opt/prometheus/node_exporter </pre>
<div class="cnblogs_code">
<pre># 创建system服务
cat > /etc/systemd/system/node_exporter.service <<"EOF"

Description=node_exporter
Documentation=https://prometheus.io/
After=network.target

User=prometheus
Group=prometheus
ExecStart=/opt/prometheus/node_exporter/node_exporter
Restart=on-failure

WantedBy=multi-user.target
EOF

# 启动服务
systemctl daemon-reload
systemctl start node_exporter.service
systemctl enable node_exporter.service
systemctl status node_exporter.service
journalctl -u node_exporter.service -f# 查看日志</pre>
</div>
<pre></pre>
服务端操作
<div class="cnblogs_code">
<pre># 修改prometheus配置
nano /opt/prometheus/prometheus/prometheus.yml

# 再scrape_configs这行下面添加如下配置：
#node-exporter配置
- job_name: 'node-exporter'
scrape_interval: 15s
static_configs:
- targets: ['192.168.10.14:9100']
 labels:
 instance: test服务器

# 重载prometheus
curl -X POST http://localhost:9090/-/reload</pre>
</div>
</div>
<h4>1.2 docker 或docker-compose安装客户端</h4>
<div class="cnblogs_code">
<pre># docker安装
docker run -d -p 9100:9100 \
-v "/proc:/host/proc:ro" \
-v "/sys:/host/sys:ro" \
-v "/:/rootfs:ro" \
--net="host" \
prom/node-exporter

# docker-compose安装
mkdir /data/node_exporter -p

cd /data/node_exporter

cat > docker-compose.yaml <<"EOF"
version: '3.3'
services:
node_exporter:
image: prom/node-exporter:v1.5.0
container_name: node-exporter
restart: always
network_mode: "host"
volumes:
 - /proc:/host/proc:ro
 - /sys:/host/sys:ro
 - /:/rootfs:ro
command:
 - '--web.listen-address=:9100'
 - '--path.procfs=/host/proc'
 - '--path.sysfs=/host/sys'
 - "--path.rootfs=/rootfs"
 - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc|rootfs/var/lib/docker)($$|/)'
EOF

# 启动和检查
docker-compose up -d

docker ps
或：
docker logs -f node-exporter http://192.168.10.100:9100/metrics</pre>
</div>
修改服务端prometheus.yml配置文件
<div class="cnblogs_code">
<pre># 增加
- job_name: 'node-exporter'
scrape_interval: 15s
static_configs:
- targets: ['node_exporter:9100']
 labels:
 instance: Prometheus服务器
- targets: ['192.168.10.100:9100']
 labels:
 instance: test服务器

# 重载服务端配置
curl -X POST http://localhost:9090/-/reload</pre>
</div>
<img src="https://img2024.cnblogs.com/blog/1523753/202404/1523753-20240424154132797-1305542463.png">
<h3> 二、常用监控指标</h3>
查看：http://192.168.10.14:9090/graph
<h4>2.1 cpu采集</h4>
node_cpu_seconds_total
<table border="0" align="left">
<tbody>
<tr>
<td>名称</td>
<td>含义</td>
</tr>
<tr>
<td>node_load1</td>
<td>1分钟内cpu负载</td>
</tr>
<tr>
<td>node_load5</td>
<td>5分钟内cpu负载</td>
</tr>
<tr>
<td>node_load15</td>
<td>15分钟内cpu负载</td>
</tr>
</tbody>
</table>
 
 
 
 
<img src="https://img2024.cnblogs.com/blog/1523753/202404/1523753-20240424155456773-1073674812.png">
<h4>2.2 内存采集/proc/meminfo文件</h4>
node_memory_
node_memory_MemTotal_bytes 内存总大小
<div class="lake-content">
node_memory_MemAvailable_bytes 空闲可使用内存大小(=free+buffer+cache)
</div>
<div class="lake-content">
node_memory_MemFree_bytes 空闲物理内存大小
</div>
<h4>2.3 磁盘采集</h4>
node_disk_
<div class="lake-content">
<div class="lake-content">
node_disk_read_bytes_total  自exporter启动以来从磁盘读取的总字节数
</div>
</div>
<div class="lake-content">
node_disk_written_bytes_total 自exporter启动以来写入到磁盘的总字节数
</div>
<h4>2.4 文件系统采集</h4>
<div class="lake-content">
node_filesystem_avail_bytes 空闲磁盘大小，单位字节  /1024/1024=MB，/1024/1024/1024=GB
</div>
<div class="lake-content">
node_filesystem_size_bytes 磁盘总大小
<div class="lake-content">
node_filesystem_files_free  空inode大小，单位个
<div class="lake-content">
node_filesystem_files inode总大小，大卫个
<h4 class="ne-p">2.5 网络采集</h4>
node_network_
<div class="lake-content">
node_network_transmit_bytes_total    网络流出流量，单位字节（Byte）/1024/1024=Mb/s
<div class="lake-content">node_network_receive_bytes_tota      网络流入流量，单位字节（Byte）</div>
</div>
</div>
</div>
</div>
2.6 文件描述符
node_filefd_allocated: 已分配的文件描述符数。通过cat /proc/sys/fs/file-nr查看 node_filefd_maximum: 系统支持的最大文件描述符数,通过/proc/sys/fs/file-max或/proc/sys/fs/file-nr
<h4>2.7 进程文件描述符</h4>
process_max_fds: 进程可打开的最大文件描述符数。 process_open_fds: node_exporter进程当前打开的文件描述符数。通过ls /proc/$PID/fd 2>/dev/null | wc -l 计算
<h4>2.8 socket</h4>
node_sockstat_sockets_used # 使用的 Socket 数 node_sockstat_TCP_inuse # 监听的 TCP Socket 数 node_sockstat_TCP_tw
<h4>2.9 TCP/UDP协议</h4>
node_netstat_Tcp_CurrEstab # ESTABLISHED 加 CLOSE_WAIT 状态的 TCP 连接数 node_netstat_Tcp_InSegs # 接收的 TCP 包数（包括错误的） node_netstat_Tcp_InErrs # 接收的 TCP 错误包数（比如校验和错误） node_netstat_Tcp_OutSegs # 发送的 TCP 包数 node_netstat_Udp_InDatagrams # 接收的 UDP 包数 node_netstat_Udp_InErrors # 接收的 UDP 错误包数 node_netstat_Udp_OutDatagrams # 发送的 UDP 包数
<h3> 三、触发器设置</h3>
<div class="cnblogs_code">
<pre>cd /data/docker-prometheus/

cat >> prometheus/alert.yml <<"EOF"
- name: node-exporter
rules:
- alert: HostOutOfMemory
expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10
for: 2m
labels:
 severity: warning
annotations:
 summary: "主机内存不足,实例:{{ $labels.instance }}"
 description: "内存可用率<10%，当前值：{{ $value }}"
- alert: HostMemoryUnderMemoryPressure
expr: rate(node_vmstat_pgmajfault) > 1000
for: 2m
labels:
 severity: warning
annotations:
 summary: "内存压力不足,实例:{{ $labels.instance }}"
 description: "节点内存压力大。重大页面错误率高，当前值为：{{ $value }}"
- alert: HostUnusualNetworkThroughputIn
expr: sum by (instance) (rate(node_network_receive_bytes_total)) / 1024 / 1024 > 100
for: 5m
labels:
 severity: warning
annotations:
 summary: "异常流入网络吞吐量,实例:{{ $labels.instance }}"
 description: "网络流入流量 > 100 MB/s，当前值：{{ $value }}"
- alert: HostUnusualNetworkThroughputOut
expr: sum by (instance) (rate(node_network_transmit_bytes_total)) / 1024 / 1024 > 100
for: 5m
labels:
 severity: warning
annotations:
 summary: "异常流出网络吞吐量，实例:{{ $labels.instance }}"
 description: "网络流出流量 > 100 MB/s，当前值为：{{ $value }}"
- alert: HostUnusualDiskReadRate
expr: sum by (instance) (rate(node_disk_read_bytes_total)) / 1024 / 1024 > 50
for: 5m
labels:
 severity: warning
annotations:
 summary: "异常磁盘读取,实例:{{ $labels.instance }}"
 description: "磁盘读取> 50 MB/s，当前值：{{ $value }}"
- alert: HostUnusualDiskWriteRate
expr: sum by (instance) (rate(node_disk_written_bytes_total)) / 1024 / 1024 > 50
for: 2m
labels:
 severity: warning
annotations:
 summary: "异常磁盘写入,实例:{{ $labels.instance }}"
 description: "磁盘写入> 50 MB/s，当前值：{{ $value }}"
- alert: HostOutOfDiskSpace
expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 10 and ON (instance, device, mountpoint) node_filesystem_readonly == 0
for: 2m
labels:
 severity: warning
annotations:
 summary: "磁盘空间不足告警,实例:{{ $labels.instance }}"
 description: "剩余磁盘空间< 10% ，当前值：{{ $value }}"
- alert: HostDiskWillFillIn24Hours
expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 10 and ON (instance, device, mountpoint) predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs"}, 24 * 3600) < 0 and ON (instance, device, mountpoint) node_filesystem_readonly == 0
for: 2m
labels:
 severity: warning
annotations:
 summary: "磁盘空间将在24小时内耗尽,实例:{{ $labels.instance }}"
 description: "以当前写入速率预计磁盘空间将在 24 小时内耗尽，当前值：{{ $value }}"
- alert: HostOutOfInodes
expr: node_filesystem_files_free{mountpoint ="/"} / node_filesystem_files{mountpoint="/"} * 100 < 10 and ON (instance, device, mountpoint) node_filesystem_readonly{mountpoint="/"} == 0
for: 2m
labels:
 severity: warning
annotations:
 summary: "磁盘Inodes不足,实例:{{ $labels.instance }}"
 description: "剩余磁盘 inodes < 10%，当前值： {{ $value }}"
- alert: HostUnusualDiskReadLatency
expr: rate(node_disk_read_time_seconds_total) / rate(node_disk_reads_completed_total) > 0.1 and rate(node_disk_reads_completed_total) > 0
for: 2m
labels:
 severity: warning
annotations:
 summary: "异常磁盘读取延迟,实例:{{ $labels.instance }}"
 description: "磁盘读取延迟 > 100ms，当前值：{{ $value }}"
- alert: HostUnusualDiskWriteLatency
expr: rate(node_disk_write_time_seconds_total) / rate(node_disk_writes_completed_total) > 0.1 and rate(node_disk_writes_completed_total) > 0
for: 2m
labels:
 severity: warning
annotations:
 summary: "异常磁盘写入延迟,实例:{{ $labels.instance }}"
 description: "磁盘写入延迟 > 100ms，当前值：{{ $value }}"
- alert: high_load
expr: node_load1 > 4
for: 2m
labels:
 severity: page
annotations:
 summary: "CPU1分钟负载过高,实例:{{ $labels.instance }}"
 description: "CPU1分钟负载>4，已经持续2分钟。当前值为：{{ $value }}"
- alert: HostCpuIsUnderUtilized
expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"})) * 100) > 80
for: 1m
labels:
 severity: warning
annotations:
 summary: "cpu负载高,实例:{{ $labels.instance }}"
 description: "cpu负载> 80%，当前值：{{ $value }}"
- alert: HostCpuStealNoisyNeighbor
expr: avg by(instance) (rate(node_cpu_seconds_total{mode="steal"})) * 100 > 10
for: 0m
labels:
 severity: warning
annotations:
 summary: "CPU窃取率异常,实例:{{ $labels.instance }}"
 description: "CPU 窃取率 > 10%。嘈杂的邻居正在扼杀 VM 性能，或者 Spot 实例可能失去信用，当前值：{{ $value }}"
- alert: HostSwapIsFillingUp
expr: (1 - (node_memory_SwapFree_bytes / node_memory_SwapTotal_bytes)) * 100 > 80
for: 2m
labels:
 severity: warning
annotations:
 summary: "磁盘swap空间使用率异常,实例:{{ $labels.instance }}"
 description: "磁盘swap空间使用率>80%"
- alert: HostNetworkReceiveErrors
expr: rate(node_network_receive_errs_total) / rate(node_network_receive_packets_total) > 0.01
for: 2m
labels:
 severity: warning
annotations:
 summary: "异常网络接收错误,实例:{{ $labels.instance }}"
 description: "网卡{{ $labels.device }}在过去2分钟接收错误率大于0.01，当前值:{{ $value }}"
- alert: HostNetworkTransmitErrors
expr: rate(node_network_transmit_errs_total) / rate(node_network_transmit_packets_total) > 0.01
for: 2m
labels:
 severity: warning
annotations:
 summary: "异常网络传输错误,实例:{{ $labels.instance }}"
 description: "网卡{{ $labels.device }}在过去2分钟传输错误率大于0.01，当前值:{{ $value }}"
- alert: HostNetworkInterfaceSaturated
expr: (rate(node_network_receive_bytes_total{device!~"^tap.*"}) + rate(node_network_transmit_bytes_total{device!~"^tap.*"})) / node_network_speed_bytes{device!~"^tap.*"} > 0.8 < 10000
for: 1m
labels:
 severity: warning
annotations:
 summary: "异常网络接口饱和,实例:{{ $labels.instance }}"
 description: "网卡{{ $labels.device }}正在超载，当前值{{ $value }}"
- alert: HostConntrackLimit
expr: node_nf_conntrack_entries / node_nf_conntrack_entries_limit > 0.8
for: 5m
labels:
 severity: warning
annotations:
 summary: "异常连接数,实例:{{ $labels.instance }}"
 description: "连接数过大，当前连接数：{{ $value }}"
- alert: HostClockSkew
expr: (node_timex_offset_seconds > 0.05 and deriv(node_timex_offset_seconds) >= 0) or (node_timex_offset_seconds < -0.05 and deriv(node_timex_offset_seconds) <= 0)
for: 2m
labels:
 severity: warning
annotations:
 summary: "异常时钟偏差,实例:{{ $labels.instance }}"
 description: "检测到时钟偏差，时钟不同步。值为：{{ $value }}"
- alert: HostClockNotSynchronising
expr: min_over_time(node_timex_sync_status) == 0 and node_timex_maxerror_seconds >= 16
for: 2m
labels:
 severity: warning
annotations:
 summary: "时钟不同步,实例:{{ $labels.instance }}"
 description: "时钟不同步"
- alert: NodeFileDescriptorLimit
expr: node_filefd_allocated / node_filefd_maximum * 100 > 80
for: 1m
labels:
 severity: warning
annotations:
 summary: "预计内核将很快耗尽文件描述符限制"
 description: "{{ $labels.instance }}｝已分配的文件描述符数超过了限制的80%，当前值为：{{ $value }}" </pre>
    - alert: tcp 已建立的连接数超过 40000       expr: node_netstat_Tcp_CurrEstab > 40000       for: 3m       labels:         severity: warning       annotations:         summary: "主机连接数过多,实例:{{ $labels.instance }}"         description: "当前值：{{ $value }}"
<pre>EOF</pre>
</div>
检查rule配置文件是否有问题：
<div class="cnblogs_code">
<pre>docker exec -it prometheus promtool check config /etc/prometheus/prometheus.yml

# 重载配置
curl -X POST http://localhost:9090/-/reload</pre>
</div>
查看：http://192.168.10.14:9090/alerts?search=，可以发现alters多了刚才定义的
<img src="https://img2024.cnblogs.com/blog/1523753/202404/1523753-20240424160057845-1275063421.png">
<h3> 四、grafana展示node-exporter数据</h3>
由于grafana已经添加了1860模版，可以直接登录查看
http://192.168.10.14:3000/login
<img src="https://img2024.cnblogs.com/blog/1523753/202404/1523753-20240424160442639-1429058656.png">
  
来源：https://www.cnblogs.com/yangmeichong/p/18155650

頁: [1]

琼殿技术论坛's Archiver

4.prometheus监控--监控linux服务器