ELK logstash 处理mongodb日志(27th)
<p>上一篇是处理MySQL的慢查询日志的，其实，ELK内容就这么多，很有规律的说，一通百通，通一反万。下面说说对mongodb日志处理。 不同mongodb版本的日志格式不同，这个需要看mongodb官方对日志格式的定义，在处理前自己去做好这方面的功课。还有就是，要抓取自己感兴趣的内容，这个根据各自的需求来做，没有千篇一律的，全凭各自喜好。</p><p>grok预定义的正则匹配规则可以参考 https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns 可以把这些文件全部下载下来放到patterns目录下，随时调用。同时，你如果安装了logstash会在这个目录下有一系列自带的正则/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.2/patterns，不同安装方式可能目录有别。</p>
<p>我这里使用版本信息如下：</p>
<ul>
<li>elasticsearch 2.2.0</li>
<li>logstash 2.2.2</li>
<li>kibana 4.4.0</li>
<li>mongodb 3.2.0</li>
<li>filebeat 1.1.1</li>
</ul>
<h3>mongodb 日志格式</h3>
<p>详细请参考 https://docs.mongodb.org/manual/reference/log-messages/#log-message-components</p>
<p>从3.0版本开始，mongodb日志内容包含severity level和component。</p><pre class="brush:bash;toolbar:false"><timestamp> <severity> <component> [<context>] <message></pre><p>如：</p><pre class="brush:bash;toolbar:false">2014-11-03T18:28:32.450-0500 I NETWORK waiting for connections on port 27017</pre><p></p>
<h4>timestamp</h4>
<p>时间戳默认使用iso8601-local</p>
<h4>severity level</h4>
<table class="docutils" border="1">
<colgroup>
<col width="15%">
<col width="85%">
</colgroup>
<thead valign="bottom"><tr class="row-odd">
<th class="head">Level</th>
<th class="head">Description</th>
</tr></thead>
<tbody valign="top">
<tr class="row-even">
<td><tt class="docutils literal"><span class="pre">F</span></tt></td>
<td>Fatal</td>
</tr>
<tr class="row-odd">
<td><tt class="docutils literal"><span class="pre">E</span></tt></td>
<td>Error</td>
</tr>
<tr class="row-even">
<td><tt class="docutils literal"><span class="pre">W</span></tt></td>
<td>Warning</td>
</tr>
<tr class="row-odd">
<td><tt class="docutils literal"><span class="pre">I</span></tt></td>
<td>Informational, for Verbosity Level of <tt class="docutils literal"><span class="pre">0</span></tt>
</td>
</tr>
<tr class="row-even">
<td><tt class="docutils literal"><span class="pre">D</span></tt></td>
<td>Debug, for All Verbosity Levels > <tt class="docutils literal"><span class="pre">0</span></tt>
</td>
</tr>
</tbody>
</table>
<h4>compoent</h4>
<table>
<thead><tr>
<th>Item</th>
<th>Description</th>
</tr></thead>
<tbody>
<tr>
<td>ACCESS</td>
<td>消息涉及到访问控制相关的，如验证。</td>
</tr>
<tr>
<td>COMMAND</td>
<td>消息涉及到数据库命令相关的，如count。</td>
</tr>
<tr>
<td>CONTROL</td>
<td>消息涉及到活动控制相关的，如 initialization。</td>
</tr>
<tr>
<td>GEO</td>
<td>消息涉及到空间地理解析相关的，如 verifying the GeoJSON shapes。</td>
</tr>
<tr>
<td>INDEX</td>
<td>消息涉及到索引操作相关的，如创建索引。</td>
</tr>
<tr>
<td>NETWORK</td>
<td>消息涉及到网络活动相关的，如接收连接。</td>
</tr>
<tr>
<td>QUERY</td>
<td>消息涉及到查询相关的，包含查询规划活动状况。</td>
</tr>
<tr>
<td>REPL</td>
<td>消息涉及到复制集相关的，如 initial sync and heartbeats。</td>
</tr>
<tr>
<td>SHARDING</td>
<td>消息涉及到分片活动相关的，如 the startup of the mongos。</td>
</tr>
<tr>
<td>STORAGE</td>
<td>消息涉及到存储活动相关的，如processes involved in the fsync command。</td>
</tr>
<tr>
<td>JOURNAL</td>
<td>消息涉及到具体journaling 活动相关的。</td>
</tr>
<tr>
<td>WRITE</td>
<td>消息涉及到写操作，如update命令。</td>
</tr>
<tr>
<td>-</td>
<td>消息不与命名组件相关的。</td>
</tr>
</tbody>
</table>
<h3>filebeat配置</h3>
<p></p><pre class="brush:bash;toolbar:false"># vi /etc/filebeat/filebeat.yml
filebeat:
prospectors:
-
paths:
- /www.ttlsa.com/logs/mysql/slow.log
document_type: mysqlslowlog
input_type: log
multiline:
negate: true
match: after
-
paths:
- /www.ttlsa.com/logs/mongodb/mongodb.log
document_type: mongodblog
registry_file: /var/lib/filebeat/registry
output:
logstash:
hosts: ["10.6.66.18:5046"]
shipper:
logging:
files:</pre><p></p>
<h3>logstash配置</h3>
<h4>1. input配置</h4>
<p>参见上一篇。</p>
<h4>2. filter配置</h4>
<p></p><pre class="brush:bash;toolbar:false"># vi 17-mongodblog.conf
filter {
if == "mongodblog" {
grok {
match => ["message","%{TIMESTAMP_ISO8601:timestamp}\s+%{MONGO3_SEVERITY:severity}\s+%{MONGO3_COMPONENT:component}\s+(?:\[%{DATA:context}\])?\s+%{GREEDYDATA:body}"]
}
if =~ "ms$"{
grok {
match => ["body","query\s+%{WORD:db_name}\.%{WORD:collection_name}.*}.*\}(\s+%{NUMBER:spend_time:int}ms$)?"]
}
}
date {
match => [ "timestamp", "UNIX", "YYYY-MM-dd HH:mm:ss", "ISO8601"]
remove_field => [ "timestamp" ]
}
}
}</pre><p></p>
<h4>3. output配置</h4>
<p></p><pre class="brush:bash;toolbar:false"># vim 30-beats-output.conf
output {
if "_grokparsefailure" in {
file { path => "/var/log/logstash/grokparsefailure-%{}-%{+YYYY.MM.dd}.log" }
}
if [@metadata] in [ "mysqlslowlog", "mongodblog" ] {
elasticsearch {
hosts => ["10.6.66.18:9200"]
sniffing => true
manage_template => false
template_overwrite => true
index => "%{[@metadata]}-%{}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata]}"
}
}</pre><p></p>
<h3>logstash 标准输出结果</h3>
<div id="attachment_11429" class="wp-caption alignnone">
<img title="ELK logstash 处理mongodb日志(27th)" class="wp-image-11429" src="https://zhuji.jb51.net/uploads/img/20230519/951eb0b95d257139d320a146f894430a.jpg" width="760" height="377"><p class="wp-caption-text">logstash-mongodb-log</p>
</div>
<h3>kibana</h3>
<div id="attachment_11430" class="wp-caption alignnone">
<img title="ELK logstash 处理mongodb日志(27th)" class="size-full wp-image-11430" src="https://zhuji.jb51.net/uploads/img/20230519/fd8b860d263b2e6756aed04d4c24c3cd.jpg" width="888" height="589"><p class="wp-caption-text">logstash-mongodb-log-kibana</p>
</div>
頁:
[1]