ELK logstash升级到2.0以及logstash-forwarder迁移到Filebeat(21st)
<p>将从logstash1.5版本升级到2.1版本，以及将《ELK部署指南》中使用的logstash-forwarder转移到Filebeat上。</p><h3>升级步骤</h3>
<ol>
<li>停止logstash以及发送到logstash的所有管道。</li>
<li>更新apt或yum源或者下载新版包。</li>
<li>安装新版的logstash。</li>
<li>测试logstash配置文件是否正确。</li>
<li>启动logstash以及第一步停止的管道。</li>
</ol>
<h3>升级logstash和elasticsearch到2.0</h3>
<p>升级前请先看看版本的改变信息。</p>
<p>下面是elasticsearch升级到2.0后，需要执行的：</p>
<p>Mapping改变：用户自定义的模板变化，因此在默认情况下，logstash升级将抛弃这些模板。即时没有一个自定义的模板，默认情况下logstash不会覆盖已存在的模板。</p>
<p>已经有一个已知的问题就是使用GeoIP过滤器需要手动更新模板。</p>
<p>注意，如果有自定义模板更改，务必保持和合并这些更改。</p>
<p>查看已有的模板：</p><pre class="brush:bash;toolbar:false">curl -XGET localhost:9200/_template/logstash</pre><p>在logstash配置文件中添加下面的配置并重启：</p><pre class="brush:bash;toolbar:false">output {
elasticsearch {
template_overwrite => true
}
}</pre><p>有点的字段：elasticsearch2.0不允许字段名含有.字符。一些插件包括logstash-filter-metrics和 logstash-filter-elapsed已经更新弥补这一更改。这些插件更新对于logstash2.0可用。要升级这些插件可执行下面命令：</p><pre class="brush:bash;toolbar:false">bin/plugin update <plugin_name></pre><p>多行过滤器：如果要在logstash配置文件中使用多行过滤器并升级到2.0，将会得到一个错误。确保filter_workers明确设置为1。如果要改变该值需通过命令行参数更改，如下所示：</p><pre class="brush:bash;toolbar:false">bin/logstash `-w 1`</pre><p><img title="ELK logstash升级到2.0以及logstash-forwarder迁移到Filebeat(21st)" class="alignnone wp-image-10825" src="https://zhuji.jb51.net/uploads/img/20230519/b31609c7be64d8984036788adbf52203.jpg" width="670" height="121"></p>
<h3>实操</h3>
<ol>
<li>关闭logstash以及输入的管道。</li>
</ol>
<p></p><pre class="brush:bash;toolbar:false"># /etc/init.d/topbeat stop
# /etc/init.d/packetbeat stop
# /etc/init.d/filebeat stop
# /etc/init.d/logstash-forwarder stop
# /etc/init.d/logstash stop</pre><p>这节会将logstash-forwarder迁移到Filebeat上了，后续不再启动logstash-forwarder了。</p>
<p>2. 升级logstash，添加yum源参见前文。</p><pre class="brush:bash;toolbar:false"># yum update logstash</pre><p>3. 检查配置文件</p>
<p>我的配置文件是以《ELK部署指南》中的配置文件为原型的。</p><pre class="brush:bash;toolbar:false"># /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/01-lumberjack-input.conf
Configuration OK
# /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/10-active.conf
Configuration OK
# /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/11-nginx.conf
Configuration OK
# /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/99-lumberjack-output.conf
Error: The setting `host` in plugin `elasticsearch` is obsolete and is no longer available. Please use the 'hosts' setting instead. You can specify multiple entries separated by comma in 'host:port' format. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask.</pre><p>更改配置文件</p><pre class="brush:bash;toolbar:false"># mv 01-lumberjack-input.conf 01-beats-input.conf
input {
beats {
port => 5044
host => "10.1.19.18"
type => "logs"
}
}</pre><p>这个是把有关logstash-forwarder使用的lumberjack删除了。</p><pre class="brush:bash;toolbar:false"># mv 99-lumberjack-output.conf 99-beats-output.conf
# vim 99-beats-output.conf
output {
if "_grokparsefailure" in {
file { path => "/var/log/logstash/grokparsefailure-%{}-%{+YYYY.MM.dd}.log" }
}
elasticsearch {
hosts => ["10.162.19.184:9200"]
sniffing => true
manage_template => false
template_overwrite => true
index => "%{[@metadata]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata]}"
}
#stdout { codec =>rubydebug }
}</pre><p>以上是output的定义。</p><pre class="brush:bash;toolbar:false"># vim 11-nginx.conf
filter {
if == "nginx" {
grok {
match => { "message" => "%{IPORHOST:clientip} - %{NOTSPACE:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:method} %{NOTSPACE:request}(?: %{URIPROTO:proto}/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} (?:%{NUMBER:upstime}|-) %{NUMBER:reqtime} (?:%{NUMBER:size}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{QS:reqbody} %{WORD:scheme} (?:%{IPV4:upstream}(:%{POSINT:port})?|-)" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
add_tag => [ "geoip" ]
fields => ["country_name", "country_code2","region_name", "city_name", "real_region_name", "latitude", "longitude"]
remove_field => [ "", "" ]
}
useragent {
source => "agent"
target => "browser"
}
}
}</pre><p>对类型nginx的filter。</p>
<p>logstash-forwarder的配置文件</p><pre class="brush:bash;toolbar:false"># vi /etc/logstash-forwarder.conf
"files": [
{
"paths": [ "/data/logs/www.ttlsa.com/active/*.log" ],
"fields": { "type": "active" }
},
{
"paths": [ "/data/logs/www.ttlsa.com/nginx/*-access.log" ],
"fields": { "type": "nginx" }
}
]</pre><p>改成Filebeat的配置文件：</p><pre class="brush:bash;toolbar:false"># vim /etc/filebeat/filebeat.yml
filebeat:
prospectors:
-
paths:
- /data/logs/www.ttlsa.com/nginx/*-access.log
document_type: nginx //隐射为type：nginx
-
paths:
- /data/logs/www.ttlsa.com/active/*.log
document_type: active</pre><p>此处document_type选项控制输出type字段，用于elasticsearch输出以确定文档类型。对于以elasticsearch输出，该值用于设置输出文档的type字段。</p>
<p>4. 检查配置文件是否正确</p><pre class="brush:bash;toolbar:false"># /opt/logstash/bin/logstash -t -f /etc/logstash/conf.d/01-beats-input.conf
Configuration OK
# /opt/logstash/bin/logstash -t -f /etc/logstash/conf.d/11-nginx.conf
Configuration OK
# /opt/logstash/bin/logstash -t -f /etc/logstash/conf.d/30-beats-output.conf
Configuration OK</pre><p>5. 启动服务</p><pre class="brush:bash;toolbar:false"># /etc/init.d/topbeat stop
# /etc/init.d/packetbeat stop
# /etc/init.d/filebeat stop
# /etc/init.d/logstash stop</pre><p>以上便是升级的过程，以及将logstash-forwarder迁移到Filebeat上了。</p>
<p>为了避免出现下面的问题：</p>
<p>sun/misc/URLClassPath.java:1003:in `getResource': java.lang.InternalError: java.io.FileNotFoundException: /alidata/server/java/jre/lib/ext/localedata.jar (Too many open files)</p>
<p>需更改下面的配置：</p><pre class="brush:bash;toolbar:false"># vim /etc/sysconfig/logstash
LS_OPEN_FILES=65535</pre><p>Error: Your application used more memory than the safety cap of 500M.</p>
<p>Specify -J-Xmx####m to increase it (#### = cap size in MB).</p>
<p>Specify -w for full OutOfMemoryError stack trace</p>
<p>需更改下面的配置：</p><pre class="brush:bash;toolbar:false"># vim /etc/sysconfig/logstash
LS_HEAP_SIZE="1024m"</pre><p></p>
頁:
[1]