ELK Packetbeat 协议和进程配置项(16th)
<p>Packetbeat 协议部分包含支持每个协议的配置项，如ports, send_request, send_response 和 特定的协议选项。</p><p>目前Packetbeat支持以下协议：</p>
<ul class="itemizedlist" type="disc">
<li class="listitem">DNS</li>
<li class="listitem">HTTP</li>
<li class="listitem">Mysql</li>
<li class="listitem">PostgreSQL</li>
<li class="listitem">Redis</li>
<li class="listitem">Thrift-RPC</li>
<li class="listitem">MongoDB</li>
<li class="listitem">Memcache</li>
</ul>
<p>配置如下：</p><pre class="brush:bash;toolbar:false">protocols:
dns:
ports:
http:
ports:
memcache:
ports:
mysql:
ports:
redis:
ports:
pgsql:
ports:
thrift:
ports: </pre><p></p>
<h3>通用协议选项</h3>
<p>下面的选项可用于所有协议：</p>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>ports</strong></p>
</div>
</div>
</div>
<p>Packetbeat以该端口捕获具体协议流量。 Packetbeat 基于这个端口安装一个BPF过滤器。如果数据包不匹配该过滤器直接丢弃。Packetbeat还使用该端口以确定每个数据包使用哪个分析器解析。</p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>send_request</strong></p>
</div>
</div>
</div>
<p>如果启用了，请求的信息(request字段)将发送到elasticsearch。默认是 false。 如果想索引整个请求该选项要启用。注意对于HTTP，请求体默认不包含，仅仅是HTTP头部。</p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>send_response</strong></p>
</div>
</div>
</div>
<p>如果启用了，响应的信息(response字段)将发送到elasticsearch。默认是 false。 如果想索引整个请求该选项要启用。注意对于HTTP，请求体默认不包含，仅仅是HTTP头部。</p>
<p>如下图</p>
<p><img title="ELK Packetbeat 协议和进程配置项(16th)" class="alignnone size-full wp-image-10764" src="https://zhuji.jb51.net/uploads/img/20230519/1241be8dcfbb73ae08c5852956046a3f.jpg" width="700" height="352"></p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>transaction_timeout</strong></p>
</div>
</div>
</div>
<p>每个协议事务超时时间。过期事务将不再关联输入响应，但是立即发送到elasticsearch。</p>
<h3>DNS配置选项</h3>
<p>配置实例如下：</p><pre class="brush:bash;toolbar:false">protocols:
dns:
ports:
# include_authorities controls whether or not the dns.authorities field
# (authority resource records) is added to messages.
# Default: false
include_authorities: true
# include_additionals controls whether or not the dns.additionals field
# (additional resource records) is added to messages.
# Default: false
include_additionals: true</pre><p></p>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>include_authorities</strong></p>
</div>
</div>
</div>
<p>如果启用， dns.authority 字段 (权威资源记录) 将加入到DNS 事件。默认false。</p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>include_additionals</strong></p>
</div>
</div>
</div>
<p>如果启用，dns.additionals字段 (额外资源记录) 将加入到DNS 事件。默认false。</p>
</div>
</div>
<div class="titlepage">
<div>
<div>
<h3 class="title">HTTP 配置选项</h3>
</div>
</div>
</div>
<p>配置实例如下：用的最多是这个，注意看</p><pre class="brush:bash;toolbar:false">protocols:
http:
# Configure the ports where to listen for HTTP traffic. You can disable
# the http protocol by commenting the list of ports.
ports:
# Uncomment the following to hide certain parameters in URL or forms attached
# to HTTP requests. The names of the parameters are case insensitive.
# The value of the parameters will be replaced with the 'xxxxx' string.
# This is generally useful for avoiding storing user passwords or other
# sensitive information.
hide_keywords: ["pass", "password", "passwd"]
# Uncomment the following to export a list of extra HTTP headers. By
default is none sent.
send_headers: ["User-Agent", "Cookie", "Set-Cookie"]
# Uncomment the following to export Cookie or Set-Cookie headers. By
# default is false.
split_coookie: true
# Configure the HTTP header that contains the real IP address.
real_ip_header: "X-Forwarded-For"</pre><p></p>
<div class="titlepage">
<div>
<div>
<p class="title"><strong>hide_keywords</strong></p>
</div>
</div>
</div>
<p>查询参数列表，Packetbeat将自动审查已存的事务。与这些参数相关的值将被替代成 <code class="literal">'xxxxx'。默认情况下，是不会更改HTTP信息的。</code></p>
<p>Packetbeat 有这个选项，是因为不同于 SQL 流量只含有密码的hash值，HTTP 流量可能包含敏感的数据。为了降低安全风险，可配置此选项避免HTTP POST发送特定的参数。</p>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>redact_authorization</strong></p>
</div>
</div>
</div>
<p>当启用该选项， Packetbeat 隐藏 <code class="literal">Authorization值和</code> <code class="literal">Proxy-Authorization </code>HTTP头部,并检查响应的字符串。</p>
<p>如果使用Basic Authentication验证，应该启用该选，因为可能包含base64未加密的用户名和密码。</p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>send_headers</strong></p>
</div>
</div>
</div>
<p>捕获一系列头部名并发送到 Elasticsearch。这些头部以json格式放在headers字典里。</p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>send_all_headers</strong></p>
</div>
</div>
</div>
<p>发送头部白名单到 Elasticsearch，如果要发送所有头部，该选项设置为TRUE。默认是false。</p>
<p><img title="ELK Packetbeat 协议和进程配置项(16th)" class="alignnone size-full wp-image-10765" src="https://zhuji.jb51.net/uploads/img/20230519/3b80cfca812f070073db0cafc4b06072.jpg" width="714" height="380"></p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<h4 class="title">
include_body_for</h4>
</div>
</div>
</div>
<p>列出内容类型列表，Packetbeat 将在response字段包含完整的HTTP有效载荷。该选项要和send_response选项一起使用。</p>
</div>
<p>如：</p><pre class="brush:bash;toolbar:false">protocols:
http:
ports:
send_response: true
include_body_for: ["text/html"]</pre><p></p>
<div class="titlepage">
<div>
<div>
<p class="title"><strong>split_cookie</strong></p>
</div>
</div>
</div>
<p><code class="literal">如果Cookie</code> 或者 <code class="literal">Set-Cookie</code> 头部被发送, 该选项控制是否把它们分割成独自值。默认false。例如，设置了该选项， HTTP 响应将成下面的json格式结果：</p><pre class="brush:bash;toolbar:false">"response": {
"code": 200,
"headers": {
"connection": "close",
"content-language": "en",
"content-type": "text/html; charset=utf-8",
"date": "Fri, 21 Nov 2014 17:07:34 GMT",
"server": "gunicorn/19.1.1",
"set-cookie": {
"csrftoken": "S9ZuJF8mvIMT5CL4T1Xqn32wkA6ZSeyf",
"expires": "Fri, 20-Nov-2015 17:07:34 GMT",
"max-age": "31449600",
"path": "/"
},
"vary": "Cookie, Accept-Language"
},
"phrase": "OK"
}</pre><p></p>
<div class="titlepage">
<div>
<div>
<p class="title"><strong>real_ip_header</strong></p>
</div>
</div>
</div>
<p>提取真实IP地址。该信息用于 <code class="literal">real_ip</code> 和 <code class="literal">client_location</code> 索引字段。</p>
<div class="titlepage">
<div>
<div>
<h3 class="title">Memcache 配置选项</h3>
</div>
</div>
</div>
<p>实例如下：</p><pre class="brush:bash;toolbar:false">memcache:
ports:
parseunknown: false
maxvalues: 0
maxbytespervalue: 100
transaction_timeout: 200
udptransactiontimeout: 200</pre><p></p>
<div class="titlepage">
<div>
<div>
<p class="title"><strong>parseunknown</strong></p>
</div>
</div>
</div>
<p>当启用该选项，迫使memcache文本协议分析器接受未知的命令。</p>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>maxvalues</strong></p>
</div>
</div>
</div>
<p>信息(multi-get)存储值的最大数量。所有值base64编码。可能值如下：</p>
<div class="itemizedlist">
<ul class="itemizedlist" type="disc">
<li class="listitem">
<code class="literal">maxvalue: -1</code>, 存储所有值 (基于文本协议 multi-get)</li>
<li class="listitem">
<code class="literal">maxvalue: 0</code>, 不存储值 (默认)</li>
<li class="listitem">
<code class="literal">maxvalue: N</code>, 最多可存 N 值</li>
</ul>
</div>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>maxbytespervalue</strong></p>
</div>
</div>
</div>
<p>复制每个值元素的最大字节数。值以base64编码，因此JSON文档中的实际大小将是maxbytespervalue的4倍。</p>
<div class="titlepage">
<div>
<div>
<p class="title"><strong>udptransactiontimeout</strong></p>
</div>
</div>
</div>
<p>事务的超时时间。默认 10000 毫秒。</p>
<div class="titlepage">
<div>
<div>
<h3 class="title">MySQL 和 PgSQL 配置选项</h3>
</div>
</div>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>max_rows</strong></p>
</div>
</div>
</div>
<p>发送到elasticsearch的SQL信息的最大行数。默认10行。</p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>max_row_length</strong></p>
</div>
</div>
</div>
<p>发送到elasticsearch的SQL信息的最大字节。默认1024 字节。</p>
<div class="titlepage">
<div>
<div>
<h3 class="title">Thrift 配置选项</h3>
</div>
</div>
</div>
<p>socket用的比较多，注意看。</p>
<p>实例如下：</p><pre class="brush:bash;toolbar:false">thrift:
transport_type: socket
protocol_type: binary
idl_files: ["tutorial.thrift", "shared.thrift"]
string_max_size: 200
collection_max_size: 20
capture_reply: true
obfuscate_strings: true
drop_after_n_struct_fields: 100</pre><p></p>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>transport_type</strong></p>
</div>
</div>
</div>
<p>Thrift 传输类型。 默认是 <code class="literal">socket。</code></p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>protocol_type</strong></p>
</div>
</div>
</div>
<p>Thrift 协议类型。 只接受二进制的TBinary协议, 这是默认的Thrift 协议。</p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>idl_files</strong></p>
</div>
</div>
</div>
<p>Thrift 接口描述语言 (IDL)文件，Packetbeat 所监控的。 可选的，因为Thrift信息包含足够的信息去解码而无需IDL文件。 然而，提供 IDL使 Packetbeat 可包含参数和异常名字。</p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>string_max_size</strong></p>
</div>
</div>
</div>
<p>参数或返回值的字符串最大长度。如果超过该值，将会被自动截断，Packetbeat在字符串后面加上个点，标示被截断。默认200。</p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>collection_max_size</strong></p>
</div>
</div>
</div>
<p>Thrift list, set, map 或者 structure 最大元素个数。如果超过这个值，Packetbeat只捕获指定元素的个数。 Packetbeat 在集合后面添加一个虚拟的元素<code class="literal">...</code> 标明被截断。默认15。</p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>capture_reply</strong></p>
</div>
</div>
</div>
<p>如果设置为false， Packetbeat 解码来自应答的方法名称，对剩余的响应信息简单的跳过。该选项对性能、磁盘使用情况或者数据保留很有用。默认TRUE。</p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>obfuscate_strings</strong></p>
</div>
</div>
</div>
<p>如果设置为TRUE， Packetbeat 将以"*"字符串取代方法参数、返回码或者异常结构发现的字符串。</p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>drop_after_n_struct_fields</strong></p>
</div>
</div>
</div>
<p>Packetbeat忽略整个事务前，可保留的最大字段数。这是个内存保护机制，不至于Packetbeat内存无限增长。因此通常设置为一个相对高的值。默认500。</p>
<div class="titlepage">
<h3 class="title">MongoDB 配置选项</h3>
<p>用的也比较多，注意看</p>
<p>实例如下：</p><pre class="brush:bash;toolbar:false">mongodb:
send_request: true
send_response: true
max_docs: 0
max_doc_length: 0</pre><p>下面两个设置用于限制Packetbeat索引中的response字段的数据量。</p>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>max_docs</strong></p>
</div>
</div>
</div>
<p>响应索引中的response字段最大的文档数。默认10。 可以设置为0，不限制文档数。</p>
<p>Packetbeat 在最后添加 <code class="literal">[...]</code> 一行标明未保存其它文档，是因为该设置。</p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>max_doc_length</strong></p>
</div>
</div>
</div>
<p>单个文档索引中的response字段的最大字符数。默认5000。 可以设置为0不限制。</p>
<p>如果因为该设置文档被修剪，Packetbeat在文档末尾添加<code class="literal">...</code> 字符串以标示。</p>
<p>注意，以该方式限制文档意味着它们将不再是正确的json格式对象了。</p>
<h3>维护网络拓扑的实时状态</h3>
<p>Packetbeat知道每个事务的源服务器和目标服务器的名称。这个之前说过的，会维护主机名、ip映射的。不清楚的看前面文章。</p>
<p>Packetbeat存储拓扑信息到elasticsearch索引。要保持网络拓扑，使用elasticsearch作为输出，并设置save_topology为TRUE。</p>
</div>
<div class="titlepage">
<div>
<div>
<h3 class="title">进程 (可选)</h3>
</div>
</div>
</div>
<p>可选的。捕获进程的通讯包。默认情况下，进程匹配是禁用的。只支持Linux系统。</p>
<p> </p><pre class="brush:bash;toolbar:false">procs:
enabled: true
monitored:
- process: mysqld
cmdline_grep: mysqld
- process: pgsql
cmdline_grep: postgres
- process: nginx
cmdline_grep: nginx
- process: app
cmdline_grep: gunicorn</pre><p></p>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>process</strong></p>
</div>
</div>
</div>
<p>指定进程名称，可随意定义以便可识。</p>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<p class="title"><strong>cmdline_grep</strong></p>
</div>
</div>
</div>
<p>该名字用于识别正在运行的进程名。定期扫描进程表与之匹配，一旦匹配从 <code class="literal">/proc/<pid>/cmdline读取信息。</code></p>
</div>
</div>
</div>
</div>
</div>
頁:
[1]