Linux下Kafka分布式集群安装教程
<p>Kafka（http://kafka.apache.org/） 是由 LinkedIn 使用 Scala 编写的一个分布式消息系统，用作 LinkedIn 的活动流（Activity Stream）和运营数据处理管道（Pipeline）的基础，具有高水平扩展和高吞吐量。Spack、Elasticsearch 都支持与 Kafka 集成。下面看一下几种分布式开源消息队列系统的对比：</p>
<p>
<img title="Linux下Kafka分布式集群安装教程" alt="Linux下Kafka分布式集群安装教程" src="https://zhuji.jb51.net/uploads/img/202305/f87ef9d3d34be698633adf92db579b42.jpg"></p>
<p>
Kafka 集群架构：</p>
<p>
<img title="Linux下Kafka分布式集群安装教程" alt="Linux下Kafka分布式集群安装教程" src="https://zhuji.jb51.net/uploads/img/202305/994062c78b99d8fb19995ab9024fd860.jpg"></p>
<p>
一般不建议直接使用 Kafka 自带的 Zookeeper 建立 zk 集群，这里我们使用独立的 zk 集群，zk 集群搭建参考文章 Linux下ZooKeeper分布式集群安装。</p>
<p>
这里 Linux 选择 CentOS 7.2。</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_907741">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash comments"># cd /usr/local</code>
</div>
<div class="line number2 index1 alt1">
<code class="bash comments"># mkdir kafka</code>
</div>
<div class="line number3 index2 alt2">
<code class="bash comments"># cd kafka</code>
</div>
<div class="line number4 index3 alt1">
<code class="bash comments"># wget -P /usr/local/kafka http://mirror.bit.edu.cn/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz</code>
</div>
<div class="line number5 index4 alt2">
<code class="bash comments"># tar -zxvf kafka_2.11-1.0.0.tgz</code>
</div>
<div class="line number6 index5 alt1">
<code class="bash comments"># mkdir kafka-logs</code>
</div>
<div class="line number7 index6 alt2">
</div>
<div class="line number8 index7 alt1">
<code class="bash comments"># cd kafka_2.11-1.0.0/config</code>
</div>
<div class="line number9 index8 alt2">
<code class="bash comments"># vim server.properties</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
编辑配置文件 (配置项可以参考 http://kafka.apache.org/documentation.html#brokerconfigs) 为：</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_172096">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
<div class="line number15 index14 alt2">
15</div>
<div class="line number16 index15 alt1">
16</div>
<div class="line number17 index16 alt2">
17</div>
<div class="line number18 index17 alt1">
18</div>
<div class="line number19 index18 alt2">
19</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash comments"># 唯一标识,0开始</code>
</div>
<div class="line number2 index1 alt1">
<code class="bash plain">broker.</code><code class="bash functions">id</code><code class="bash plain">=0</code>
</div>
<div class="line number3 index2 alt2">
</div>
<div class="line number4 index3 alt1">
<code class="bash comments"># 服务器端口和ip</code>
</div>
<div class="line number5 index4 alt2">
<code class="bash plain">port=9092</code>
</div>
<div class="line number6 index5 alt1">
<code class="bash plain">host.name=192.168.2.20</code>
</div>
<div class="line number7 index6 alt2">
<code class="bash comments"># 日志数据保存的目录</code>
</div>
<div class="line number8 index7 alt1">
<code class="bash plain">log.</code><code class="bash functions">dirs</code><code class="bash plain">=</code><code class="bash plain">/usr/local/kafka/kafka-logs</code>
</div>
<div class="line number9 index8 alt2">
</div>
<div class="line number10 index9 alt1">
<code class="bash plain">log.retention.hours=168</code>
</div>
<div class="line number11 index10 alt2">
<code class="bash comments"># 每条消息能够容纳的最大大小,默认1000012</code>
</div>
<div class="line number12 index11 alt1">
<code class="bash plain">message.max.bytes=5242880</code>
</div>
<div class="line number13 index12 alt2">
<code class="bash comments"># 默认的副本因子,即保存消息的副本数,默认1</code>
</div>
<div class="line number14 index13 alt1">
<code class="bash plain">default.replication.factor=2</code>
</div>
<div class="line number15 index14 alt2">
<code class="bash comments"># 取消息的最大直接数,默认1048576</code>
</div>
<div class="line number16 index15 alt1">
<code class="bash plain">replica.fetch.max.bytes=5242880</code>
</div>
<div class="line number17 index16 alt2">
</div>
<div class="line number18 index17 alt1">
<code class="bash comments"># 配置zk地址</code>
</div>
<div class="line number19 index18 alt2">
<code class="bash plain">zookeeper.connect=192.168.20.11:2181,192.168.20.12:2181,192.168.20.13:2181</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
启动 Kafka 之前需启动 Zookeeper，然后启动 Kafka，各个节点需单独启动：</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_771229">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash comments"># cd /usr/local/kafka/kafka_2.11-1.0.0/bin</code>
</div>
<div class="line number2 index1 alt1">
<code class="bash comments"># ./kafka-server-start.sh -daemon ../config/server.properties //后台启动服务</code>
</div>
<div class="line number3 index2 alt2">
<code class="bash comments"># jps //查看服务是否启动成功</code>
</div>
<div class="line number4 index3 alt1">
<code class="bash comments"># ./kafka-server-stop.sh //关闭服务</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
最后我们在三台不同的服务器上各部署一个 Kafka 进程，以构成一个 Kafka 集群。开始使用可以参考官方文档 http://kafka.apache.org/documentation.html#quickstart。</p>
<p>
以上就是本文的全部内容，希望对大家的学习有所帮助，也希望大家多多支持。</p>
<p>
原文链接：http://blog.csdn.net/smartbetter/article/details/64190798</p>
頁:
[1]