基于CentOS的Hadoop分布式环境的搭建开发

以德报德 發表於 2023-9-22 00:00:00

基于CentOS的Hadoop分布式环境的搭建开发

<p>
首先，要说明的一点的是，我不想重复发明轮子。如果想要搭建hadoop环境，网上有很多详细的步骤和命令代码，我不想再重复记录。</p>
<p>
其次，我要说的是我也是新手，对于hadoop也不是很熟悉。但是就是想实际搭建好环境，看看他的庐山真面目，还好，还好，最好看到了。当运行wordcount词频统计的时候，实在是感叹hadoop已经把分布式做的如此之好，即使没有分布式相关经验的人，也只需要做一些配置即可运行分布式集群环境。</p>
<p>
好了，言归真传。</p>
<p>
<strong>在搭建hadoop环境中你要知道的一些事儿：</strong></p>
<p>
1.hadoop运行于linux系统之上，你要安装linux操作系统</p>
<p>
2.你需要搭建一个运行hadoop的集群，例如局域网内能互相访问的linux系统</p>
<p>
3.为了实现集群之间的相互访问，你需要做到ssh无密钥登录</p>
<p>
4.hadoop的运行在jvm上的，也就是说你需要安装java的jdk，并配置好java_home</p>
<p>
5.hadoop的各个组件是通过xml来配置的。在官网上下载好hadoop之后解压缩，修改/etc/hadoop目录中相应的配置文件</p>
<p>
工欲善其事，必先利其器。这里也要说一下，<strong>在搭建hadoop环境中使用到的相关软件和工具：</strong></p>
<p>
1.virtualbox——毕竟要模拟几台linux，条件有限，就在virtualbox中创建几台虚拟机楼</p>
<p>
2.centos——下载的centos7的iso镜像，加载到virtualbox中，安装运行</p>
<p>
3.securecrt——可以ssh远程访问linux的软件</p>
<p>
4.winscp——实现windows和linux的通信</p>
<p>
5.jdk for linux——oracle官网上下载，解压缩之后配置一下即可</p>
<p>
6.hadoop2.7.1——可在apache官网上下载</p>
<p>
好了，下面分三个步骤来讲解</p>
<p>
<strong>linux环境准备</strong></p>
<p>
<strong> 配置ip</strong></p>
<p>
为了实现本机和虚拟机以及虚拟机和虚拟机之间的通信，virtualbox中设置centos的连接模式为host-only模式，并且手动设置ip，注意虚拟机的网关和本机中host-only network 的ip地址相同。配置ip完成后还要重启网络服务以使得配置有效。这里搭建了三台linux，如下图所示</p>
<p>
<img style="max-width:100%!important;height:auto!important;"title="基于CentOS的Hadoop分布式环境的搭建开发" alt="基于CentOS的Hadoop分布式环境的搭建开发" height="487" src="https://zhuji.jb51.net/uploads/img/202305/7726b6c1c020beb2b0511d8a8bd8bb83.jpg" width="774"></p>
<p>
<img style="max-width:100%!important;height:auto!important;"title="基于CentOS的Hadoop分布式环境的搭建开发" alt="基于CentOS的Hadoop分布式环境的搭建开发" height="557" src="https://zhuji.jb51.net/uploads/img/202305/cb8638fbc56c15beff1297a7e2561c47.jpg" width="780"></p>
<p>
<img style="max-width:100%!important;height:auto!important;"title="基于CentOS的Hadoop分布式环境的搭建开发" alt="基于CentOS的Hadoop分布式环境的搭建开发" height="556" src="https://zhuji.jb51.net/uploads/img/202305/99e4373cf700016c290fcff5991f56c2.jpg" width="769"></p>
<p>
<img style="max-width:100%!important;height:auto!important;"title="基于CentOS的Hadoop分布式环境的搭建开发" alt="基于CentOS的Hadoop分布式环境的搭建开发" height="253" src="https://zhuji.jb51.net/uploads/img/202305/681743b1ecbd94db67b7d4a6e2d68cdb.jpg" width="773"></p>
<p>
配置主机名字</p>
<p>
对于192.168.56.101设置主机名字hadoop01。并在hosts文件中配置集群的ip和主机名。其余两个主机的操作与此类似</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_13527">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain"></code><code class="bash comments"># cat /etc/sysconfig/network </code>
</div>
<div class="line number2 index1 alt1">
<code class="bash comments"># created by anaconda </code>
</div>
<div class="line number3 index2 alt2">
<code class="bash plain">networking = </code><code class="bash functions">yes</code>
</div>
<div class="line number4 index3 alt1">
<code class="bash functions">hostname</code> <code class="bash plain">= hadoop01   </code>
</div>
<div class="line number5 index4 alt2">
<code class="bash plain"></code><code class="bash comments"># cat /etc/hosts </code>
</div>
<div class="line number6 index5 alt1">
<code class="bash plain">127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 </code>
</div>
<div class="line number7 index6 alt2">
<code class="bash plain">::1     localhost localhost.localdomain localhost6 localhost6.localdomain6 </code>
</div>
<div class="line number8 index7 alt1">
<code class="bash plain">192.168.56.101 hadoop01 </code>
</div>
<div class="line number9 index8 alt2">
<code class="bash plain">192.168.56.102 hadoop02 </code>
</div>
<div class="line number10 index9 alt1">
<code class="bash plain">192.168.56.103 hadoop03</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
<strong>永久关闭防火墙 </strong></p>
<p>
service iptables stop(1.下次重启机器后，防火墙又会启动，故需要永久关闭防火墙的命令；2由于用的是centos 7,关闭防火墙的命令如下）</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_161713">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain">systemctl stop firewalld.service    </code><code class="bash comments">#停止firewall</code>
</div>
<div class="line number2 index1 alt1">
<code class="bash plain">systemctl disable firewalld.service </code><code class="bash comments">#禁止firewall开机启动</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
<strong>关闭selinux防护系统</strong></p>
<p>
改为disabled 。reboot重启机器，使配置生效</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_431709">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain"></code><code class="bash comments"># cat /etc/sysconfig/selinux </code>
</div>
<div class="line number2 index1 alt1">
<code class="bash spaces"> </code>
</div>
<div class="line number3 index2 alt2">
<code class="bash comments"># this file controls the state of selinux on the system </code>
</div>
<div class="line number4 index3 alt1">
<code class="bash comments"># selinux= can take one of these three values: </code>
</div>
<div class="line number5 index4 alt2">
<code class="bash comments">#   enforcing - selinux security policy is enforced </code>
</div>
<div class="line number6 index5 alt1">
<code class="bash spaces"> </code>
</div>
<div class="line number7 index6 alt2">
<code class="bash comments">#   permissive - selinux prints warnings instead of enforcing </code>
</div>
<div class="line number8 index7 alt1">
<code class="bash comments">#   disabled - no selinux policy is loaded </code>
</div>
<div class="line number9 index8 alt2">
<code class="bash plain">selinux=disabled </code>
</div>
<div class="line number10 index9 alt1">
<code class="bash comments"># selinuxtype= can take one of three two values: </code>
</div>
<div class="line number11 index10 alt2">
<code class="bash comments">#   targeted - targeted processes are protected, </code>
</div>
<div class="line number12 index11 alt1">
<code class="bash comments">#   minimum - modification of targeted policy only selected processes are protected </code>
</div>
<div class="line number13 index12 alt2">
<code class="bash comments">#   mls - multi level security protection </code>
</div>
<div class="line number14 index13 alt1">
<code class="bash plain">selinuxtype=targeted</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
<strong>集群ssh免密码登录</strong></p>
<p>
首先设置ssh密钥</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_484606">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash functions">ssh</code><code class="bash plain">-keygen -t rsa </code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
拷贝ssh密钥到三台机器</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_638765">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash functions">ssh</code><code class="bash plain">-copy-</code><code class="bash functions">id</code> <code class="bash plain">192.168.56.101 </code>
</div>
<div class="line number2 index1 alt1">
<code class="bash plain"><pre name=</code><code class="bash string">"code"</code> <code class="bash plain">class=</code><code class="bash string">"plain"</code><code class="bash plain">></code><code class="bash functions">ssh</code><code class="bash plain">-copy-</code><code class="bash functions">id</code> <code class="bash plain">192.168.56.102</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_945504">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash functions">ssh</code><code class="bash plain">-copy-</code><code class="bash functions">id</code> <code class="bash plain">192.168.56.103</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
这样如果hadoop01的机器想要登录hadoop02，直接输入ssh hadoop02</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_179853">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain"><pre name=</code><code class="bash string">"code"</code> <code class="bash plain">class=</code><code class="bash string">"plain"</code><code class="bash plain">></code><code class="bash functions">ssh</code> <code class="bash plain">hadoop02</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
<strong>配置jdk</strong></p>
<p>
这里在/home忠诚创建三个文件夹中</p>
<p>
tools——存放工具包</p>
<p>
softwares——存放软件</p>
<p>
data——存放数据</p>
<p>
通过winscp将下载好的linux jdk上传到hadoop01的/home/tools中</p>
<p>
解压缩jdk到softwares中</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_560375">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain"><pre name=</code><code class="bash string">"code"</code> <code class="bash plain">class=</code><code class="bash string">"plain"</code><code class="bash plain">></code><code class="bash functions">tar</code> <code class="bash plain">-zxf jdk-7u76-linux-x64.</code><code class="bash functions">tar</code><code class="bash plain">.gz -c </code><code class="bash plain">/home/softwares</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
可见jdk的家目录在/home/softwares/jdk.x.x.x，将该目录拷贝粘贴到/etc/profile文件中，并且在文件中设置java_home</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_363586">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash functions">export</code> <code class="bash plain">java_home=</code><code class="bash plain">/home/softwares/jdk0_111</code>
</div>
<div class="line number2 index1 alt1">
<code class="bash functions">export</code> <code class="bash plain">path=$path:$java_home</code><code class="bash plain">/bin</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
保存修改,执行source /etc/profile使配置生效</p>
<p>
查看java jdk是否安装成功:</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_58409">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain">java -version</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
可以将当前节点中设置的文件拷贝到其他节点</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_349118">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash functions">scp</code> <code class="bash plain">-r </code><code class="bash plain">/home/</code><code class="bash plain">* root@192.168.56.10x:</code><code class="bash plain">/home</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
<strong>hadoop集群安装</strong></p>
<p>
集群的规划如下：</p>
<p>
101节点作为hdfs的namenode ,其余作为datanode;102作为yarn的resourcemanager，其余作为nodemanager。103作为secondarynamenode。分别在101和102节点启动jobhistoryserver和webappproxyserver<img style="max-width:100%!important;height:auto!important;"title="基于CentOS的Hadoop分布式环境的搭建开发" alt="基于CentOS的Hadoop分布式环境的搭建开发" src="https://zhuji.jb51.net/uploads/img/202305/04765b334c58208dc18560e649372b9c.jpg"></p>
<p>
<strong>下载hadoop-2.7.3</strong></p>
<p>
并将其放在/home/softwares文件夹中。由于hadoop需要jdk的安装环境，所以首先配置/etc/hadoop/hadoop-env.sh的java_home</p>
<p>
（ps：感觉我用的jdk版本过高了）<img style="max-width:100%!important;height:auto!important;"title="基于CentOS的Hadoop分布式环境的搭建开发" alt="基于CentOS的Hadoop分布式环境的搭建开发" src="https://zhuji.jb51.net/uploads/img/202305/59303b62e45dce57dadfacd8b1295046.jpg"></p>
<p>
接下来依次修改hadoop相应组件对应的xml</p>
<p>
修改core-site.xml ：</p>
<p>
指定namenode地址</p>
<p>
修改hadoop的缓存目录</p>
<p>
hadoop的垃圾回收机制</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterxml" id="highlighter_322800">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
<div class="line number15 index14 alt2">
15</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="xml plain"><</code><code class="xml keyword">configuration</code><code class="xml plain">> </code>
</div>
<div class="line number2 index1 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number3 index2 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>fsdefaultfs</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number4 index3 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>hdfs://101:8020</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number5 index4 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number6 index5 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number7 index6 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>hadooptmpdir</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number8 index7 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>/home/softwares/hadoop-3/data/tmp</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number9 index8 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number10 index9 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number11 index10 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>fstrashinterval</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number12 index11 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>10080</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number13 index12 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number14 index13 alt1">
<code class="xml spaces">   </code>
</div>
<div class="line number15 index14 alt2">
<code class="xml plain"></</code><code class="xml keyword">configuration</code><code class="xml plain">></code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
hdfs-site.xml</p>
<p>
设置备份数目</p>
<p>
关闭权限</p>
<p>
设置http访问接口</p>
<p>
设置secondary namenode 的ip地址</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterxml" id="highlighter_754681">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
<div class="line number15 index14 alt2">
15</div>
<div class="line number16 index15 alt1">
16</div>
<div class="line number17 index16 alt2">
17</div>
<div class="line number18 index17 alt1">
18</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="xml plain"><</code><code class="xml keyword">configuration</code><code class="xml plain">> </code>
</div>
<div class="line number2 index1 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number3 index2 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>dfsreplication</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number4 index3 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>3</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number5 index4 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number6 index5 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number7 index6 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>dfspermissionsenabled</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number8 index7 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>false</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number9 index8 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number10 index9 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number11 index10 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>dfsnamenodehttp-address</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number12 index11 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>101:50070</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number13 index12 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number14 index13 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number15 index14 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>dfsnamenodesecondaryhttp-address</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number16 index15 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>103:50090</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number17 index16 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number18 index17 alt1">
<code class="xml plain"></</code><code class="xml keyword">configuration</code><code class="xml plain">> </code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
修改mapred-site.xml.template名字为mapred-site.xml</p>
<p>
指定mapreduce的框架为yarn，通过yarn来调度</p>
<p>
指定jobhitory</p>
<p>
指定jobhitory的web端口</p>
<p>
开启uber模式——这是针对mapreduce的优化</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterxml" id="highlighter_652228">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
<div class="line number15 index14 alt2">
15</div>
<div class="line number16 index15 alt1">
16</div>
<div class="line number17 index16 alt2">
17</div>
<div class="line number18 index17 alt1">
18</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="xml plain"><</code><code class="xml keyword">configuration</code><code class="xml plain">> </code>
</div>
<div class="line number2 index1 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number3 index2 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>mapreduceframeworkname</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number4 index3 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>yarn</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number5 index4 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number6 index5 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number7 index6 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>mapreducejobhistoryaddress</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number8 index7 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>101:10020</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number9 index8 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number10 index9 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number11 index10 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>mapreducejobhistorywebappaddress</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number12 index11 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>101:19888</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number13 index12 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number14 index13 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number15 index14 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>mapreducejobubertaskenable</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number16 index15 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>true</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number17 index16 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number18 index17 alt1">
<code class="xml plain"></</code><code class="xml keyword">configuration</code><code class="xml plain">></code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
修改yarn-site.xml</p>
<p>
指定mapreduce为shuffle</p>
<p>
指定102节点为resourcemanager</p>
<p>
指定102节点的安全代理</p>
<p>
开启yarn的日志</p>
<p>
指定yarn日志删除时间</p>
<p>
指定nodemanager的内存：8g</p>
<p>
指定nodemanager的cpu：8核</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterxml" id="highlighter_18958">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
<div class="line number15 index14 alt2">
15</div>
<div class="line number16 index15 alt1">
16</div>
<div class="line number17 index16 alt2">
17</div>
<div class="line number18 index17 alt1">
18</div>
<div class="line number19 index18 alt2">
19</div>
<div class="line number20 index19 alt1">
20</div>
<div class="line number21 index20 alt2">
21</div>
<div class="line number22 index21 alt1">
22</div>
<div class="line number23 index22 alt2">
23</div>
<div class="line number24 index23 alt1">
24</div>
<div class="line number25 index24 alt2">
25</div>
<div class="line number26 index25 alt1">
26</div>
<div class="line number27 index26 alt2">
27</div>
<div class="line number28 index27 alt1">
28</div>
<div class="line number29 index28 alt2">
29</div>
<div class="line number30 index29 alt1">
30</div>
<div class="line number31 index30 alt2">
31</div>
<div class="line number32 index31 alt1">
32</div>
<div class="line number33 index32 alt2">
33</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="xml plain"><</code><code class="xml keyword">configuration</code><code class="xml plain">> </code>
</div>
<div class="line number2 index1 alt1">
<code class="xml spaces"> </code>
</div>
<div class="line number3 index2 alt2">
<code class="xml comments"></code>
</div>
<div class="line number4 index3 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number5 index4 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>yarnnodemanageraux-services</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number6 index5 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>mapreduce_shuffle</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number7 index6 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number8 index7 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number9 index8 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>yarnresourcemanagerhostname</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number10 index9 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>102</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number11 index10 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number12 index11 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number13 index12 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>yarnweb-proxyaddress</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number14 index13 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>102:8888</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number15 index14 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number16 index15 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number17 index16 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>yarnlog-aggregation-enable</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number18 index17 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>true</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number19 index18 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number20 index19 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number21 index20 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>yarnlog-aggregationretain-seconds</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number22 index21 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>604800</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number23 index22 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number24 index23 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number25 index24 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>yarnnodemanagerresourcememory-mb</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number26 index25 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>8192</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number27 index26 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number28 index27 alt1">
<code class="xml spaces">  </code><code class="xml plain"><</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number29 index28 alt2">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">name</code><code class="xml plain">>yarnnodemanagerresourcecpu-vcores</</code><code class="xml keyword">name</code><code class="xml plain">> </code>
</div>
<div class="line number30 index29 alt1">
<code class="xml spaces">    </code><code class="xml plain"><</code><code class="xml keyword">value</code><code class="xml plain">>8</</code><code class="xml keyword">value</code><code class="xml plain">> </code>
</div>
<div class="line number31 index30 alt2">
<code class="xml spaces">  </code><code class="xml plain"></</code><code class="xml keyword">property</code><code class="xml plain">> </code>
</div>
<div class="line number32 index31 alt1">
<code class="xml spaces"> </code>
</div>
<div class="line number33 index32 alt2">
<code class="xml plain"></</code><code class="xml keyword">configuration</code><code class="xml plain">></code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
配置slaves</p>
<p>
指定计算节点，即运行datanode和nodemanager的节点</p>
<p>
192.168.56.101 <br>
192.168.56.102 <br>
192.168.56.103 </p>
<p>
先在namenode节点格式化，即101节点上执行：</p>
<p>
进入到hadoop主目录： cd /home/softwares/hadoop-3  </p>
<p>
执行bin目录下的hadoop脚本： bin/hadoop namenode -format </p>
<p>
出现successful format才算是执行成功（ps，这里是盗用别人的图，不要介意哈） <img style="max-width:100%!important;height:auto!important;"title="基于CentOS的Hadoop分布式环境的搭建开发" alt="基于CentOS的Hadoop分布式环境的搭建开发" src="https://zhuji.jb51.net/uploads/img/202305/3b8e76b28df787f099b150b85c9a4541.jpg"></p>
<p>
以上配置完成后，将其拷贝到其他的机器</p>
<p>
<strong>hadoop环境测试</strong></p>
<p>
进入hadoop主目录下执行相应的脚本文件</p>
<p>
jps命令——java virtual machine process status，显示运行的java进程</p>
<p>
<strong>在namenode节点101机器上开启hdfs</strong></p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_279671">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain"></code><code class="bash comments"># sbin/start-dfssh </code>
</div>
<div class="line number2 index1 alt1">
<code class="bash plain">java hotspot(tm) client vm warning: you have loaded library </code><code class="bash plain">/home/softwares/hadoop-3/lib/native/libhadoopso</code> <code class="bash functions">which</code> <code class="bash plain">might have disabled stack guard the vm will try to fix the stack guard now </code>
</div>
<div class="line number3 index2 alt2">
<code class="bash plain">it</code><code class="bash string">'s highly recommended that you fix the library with '</code><code class="bash plain">execstack -c <libfile></code><code class="bash string">', or link it with '</code><code class="bash plain">-z noexecstack' </code>
</div>
<div class="line number4 index3 alt1">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">16:49:19 warn utilnativecodeloader: unable to load native-hadoop library </code><code class="bash keyword">for</code> <code class="bash plain">your platform using </code><code class="bash functions">builtin</code><code class="bash plain">-java classes where applicable </code>
</div>
<div class="line number5 index4 alt2">
<code class="bash plain">starting namenodes on </code>
</div>
<div class="line number6 index5 alt1">
<code class="bash plain">hadoop01: starting namenode, logging to </code><code class="bash plain">/home/softwares/hadoop-3/logs/hadoop-root-namenode-hadoopout</code>
</div>
<div class="line number7 index6 alt2">
<code class="bash plain">102: starting datanode, logging to </code><code class="bash plain">/home/softwares/hadoop-3/logs/hadoop-root-datanode-hadoopout</code>
</div>
<div class="line number8 index7 alt1">
<code class="bash plain">103: starting datanode, logging to </code><code class="bash plain">/home/softwares/hadoop-3/logs/hadoop-root-datanode-hadoopout</code>
</div>
<div class="line number9 index8 alt2">
<code class="bash plain">101: starting datanode, logging to </code><code class="bash plain">/home/softwares/hadoop-3/logs/hadoop-root-datanode-hadoopout</code>
</div>
<div class="line number10 index9 alt1">
<code class="bash plain">starting secondary namenodes </code>
</div>
<div class="line number11 index10 alt2">
<code class="bash plain">hadoop03: starting secondarynamenode, logging to </code><code class="bash plain">/home/softwares/hadoop-3/logs/hadoop-root-secondarynamenode-hadoopout</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
此时101节点上执行jps，可以看到namenode和datanode已经启动</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_872694">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain"></code><code class="bash comments"># jps </code>
</div>
<div class="line number2 index1 alt1">
<code class="bash plain">7826 jps </code>
</div>
<div class="line number3 index2 alt2">
<code class="bash plain">7270 datanode </code>
</div>
<div class="line number4 index3 alt1">
<code class="bash plain">7052 namenode</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
在102和103节点执行jps，则可以看到datanode已经启动</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_255203">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain"></code><code class="bash comments"># jps </code>
</div>
<div class="line number2 index1 alt1">
<code class="bash plain">4260 datanode </code>
</div>
<div class="line number3 index2 alt2">
<code class="bash plain">4488 jps </code>
</div>
<div class="line number4 index3 alt1">
<code class="bash spaces"> </code>
</div>
<div class="line number5 index4 alt2">
<code class="bash plain"></code><code class="bash comments"># jps </code>
</div>
<div class="line number6 index5 alt1">
<code class="bash plain">6436 secondarynamenode </code>
</div>
<div class="line number7 index6 alt2">
<code class="bash plain">6750 jps </code>
</div>
<div class="line number8 index7 alt1">
<code class="bash plain">6191 datanode</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
启动yarn</p>
<p>
在102节点执行</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_133504">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain"></code><code class="bash comments"># sbin/start-yarnsh </code>
</div>
<div class="line number2 index1 alt1">
<code class="bash plain">starting yarn daemons </code>
</div>
<div class="line number3 index2 alt2">
<code class="bash plain">starting resourcemanager, logging to </code><code class="bash plain">/home/softwares/hadoop-3/logs/yarn-root-resourcemanager-hadoopout</code>
</div>
<div class="line number4 index3 alt1">
<code class="bash plain">101: starting nodemanager, logging to </code><code class="bash plain">/home/softwares/hadoop-3/logs/yarn-root-nodemanager-hadoopout</code>
</div>
<div class="line number5 index4 alt2">
<code class="bash plain">103: starting nodemanager, logging to </code><code class="bash plain">/home/softwares/hadoop-3/logs/yarn-root-nodemanager-hadoopout</code>
</div>
<div class="line number6 index5 alt1">
<code class="bash plain">102: starting nodemanager, logging to </code><code class="bash plain">/home/softwares/hadoop-3/logs/yarn-root-nodemanager-hadoopout</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
jps查看各节点：</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_58617">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
<div class="line number15 index14 alt2">
15</div>
<div class="line number16 index15 alt1">
16</div>
<div class="line number17 index16 alt2">
17</div>
<div class="line number18 index17 alt1">
18</div>
<div class="line number19 index18 alt2">
19</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain"></code><code class="bash comments"># jps </code>
</div>
<div class="line number2 index1 alt1">
<code class="bash plain">4641 resourcemanager </code>
</div>
<div class="line number3 index2 alt2">
<code class="bash plain">4260 datanode </code>
</div>
<div class="line number4 index3 alt1">
<code class="bash plain">4765 nodemanager </code>
</div>
<div class="line number5 index4 alt2">
<code class="bash plain">5165 jps </code>
</div>
<div class="line number6 index5 alt1">
<code class="bash spaces"> </code>
</div>
<div class="line number7 index6 alt2">
<code class="bash spaces"> </code>
</div>
<div class="line number8 index7 alt1">
<code class="bash plain"></code><code class="bash comments"># jps </code>
</div>
<div class="line number9 index8 alt2">
<code class="bash plain">7270 datanode </code>
</div>
<div class="line number10 index9 alt1">
<code class="bash plain">8375 jps </code>
</div>
<div class="line number11 index10 alt2">
<code class="bash plain">7976 nodemanager </code>
</div>
<div class="line number12 index11 alt1">
<code class="bash plain">7052 namenode </code>
</div>
<div class="line number13 index12 alt2">
<code class="bash spaces"> </code>
</div>
<div class="line number14 index13 alt1">
<code class="bash spaces"> </code>
</div>
<div class="line number15 index14 alt2">
<code class="bash plain"></code><code class="bash comments"># jps </code>
</div>
<div class="line number16 index15 alt1">
<code class="bash plain">6915 nodemanager </code>
</div>
<div class="line number17 index16 alt2">
<code class="bash plain">6436 secondarynamenode </code>
</div>
<div class="line number18 index17 alt1">
<code class="bash plain">7287 jps </code>
</div>
<div class="line number19 index18 alt2">
<code class="bash plain">6191 datanode</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
分别启动相应节点的jobhistory和防护进程</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_650334">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
<div class="line number15 index14 alt2">
15</div>
<div class="line number16 index15 alt1">
16</div>
<div class="line number17 index16 alt2">
17</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain"></code><code class="bash comments"># sbin/mr-jobhistory-daemonsh start historyserver </code>
</div>
<div class="line number2 index1 alt1">
<code class="bash plain">starting historyserver, logging to </code><code class="bash plain">/home/softwares/hadoop-3/logs/mapred-root-historyserver-hadoopout</code>
</div>
<div class="line number3 index2 alt2">
<code class="bash plain"></code><code class="bash comments"># jps </code>
</div>
<div class="line number4 index3 alt1">
<code class="bash plain">8624 jps </code>
</div>
<div class="line number5 index4 alt2">
<code class="bash plain">7270 datanode </code>
</div>
<div class="line number6 index5 alt1">
<code class="bash plain">7976 nodemanager </code>
</div>
<div class="line number7 index6 alt2">
<code class="bash plain">8553 jobhistoryserver </code>
</div>
<div class="line number8 index7 alt1">
<code class="bash plain">7052 namenode </code>
</div>
<div class="line number9 index8 alt2">
<code class="bash spaces"> </code>
</div>
<div class="line number10 index9 alt1">
<code class="bash plain"></code><code class="bash comments"># sbin/yarn-daemonsh start proxyserver </code>
</div>
<div class="line number11 index10 alt2">
<code class="bash plain">starting proxyserver, logging to </code><code class="bash plain">/home/softwares/hadoop-3/logs/yarn-root-proxyserver-hadoopout</code>
</div>
<div class="line number12 index11 alt1">
<code class="bash plain"></code><code class="bash comments"># jps </code>
</div>
<div class="line number13 index12 alt2">
<code class="bash plain">4641 resourcemanager </code>
</div>
<div class="line number14 index13 alt1">
<code class="bash plain">4260 datanode </code>
</div>
<div class="line number15 index14 alt2">
<code class="bash plain">5367 webappproxyserver </code>
</div>
<div class="line number16 index15 alt1">
<code class="bash plain">5402 jps </code>
</div>
<div class="line number17 index16 alt2">
<code class="bash plain">4765 nodemanager</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
在hadoop01节点，即101节点上，通过浏览器查看节点状况 <img style="max-width:100%!important;height:auto!important;"title="基于CentOS的Hadoop分布式环境的搭建开发" alt="基于CentOS的Hadoop分布式环境的搭建开发" src="https://zhuji.jb51.net/uploads/img/202305/f17f4c2f958c4f52c79493d3a0e7bd30.jpg"><img style="max-width:100%!important;height:auto!important;"title="基于CentOS的Hadoop分布式环境的搭建开发" alt="基于CentOS的Hadoop分布式环境的搭建开发" src="https://zhuji.jb51.net/uploads/img/202305/dfbd815f073239cea2747742dd2f330e.jpg"></p>
<p>
<strong>hdfs上传文件</strong></p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_445294">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain"></code><code class="bash comments"># bin/hdfs dfs -put /etc/profile /profile</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
运行wordcount程序</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_226657">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
<div class="line number15 index14 alt2">
15</div>
<div class="line number16 index15 alt1">
16</div>
<div class="line number17 index16 alt2">
17</div>
<div class="line number18 index17 alt1">
18</div>
<div class="line number19 index18 alt2">
19</div>
<div class="line number20 index19 alt1">
20</div>
<div class="line number21 index20 alt2">
21</div>
<div class="line number22 index21 alt1">
22</div>
<div class="line number23 index22 alt2">
23</div>
<div class="line number24 index23 alt1">
24</div>
<div class="line number25 index24 alt2">
25</div>
<div class="line number26 index25 alt1">
26</div>
<div class="line number27 index26 alt2">
27</div>
<div class="line number28 index27 alt1">
28</div>
<div class="line number29 index28 alt2">
29</div>
<div class="line number30 index29 alt1">
30</div>
<div class="line number31 index30 alt2">
31</div>
<div class="line number32 index31 alt1">
32</div>
<div class="line number33 index32 alt2">
33</div>
<div class="line number34 index33 alt1">
34</div>
<div class="line number35 index34 alt2">
35</div>
<div class="line number36 index35 alt1">
36</div>
<div class="line number37 index36 alt2">
37</div>
<div class="line number38 index37 alt1">
38</div>
<div class="line number39 index38 alt2">
39</div>
<div class="line number40 index39 alt1">
40</div>
<div class="line number41 index40 alt2">
41</div>
<div class="line number42 index41 alt1">
42</div>
<div class="line number43 index42 alt2">
43</div>
<div class="line number44 index43 alt1">
44</div>
<div class="line number45 index44 alt2">
45</div>
<div class="line number46 index45 alt1">
46</div>
<div class="line number47 index46 alt2">
47</div>
<div class="line number48 index47 alt1">
48</div>
<div class="line number49 index48 alt2">
49</div>
<div class="line number50 index49 alt1">
50</div>
<div class="line number51 index50 alt2">
51</div>
<div class="line number52 index51 alt1">
52</div>
<div class="line number53 index52 alt2">
53</div>
<div class="line number54 index53 alt1">
54</div>
<div class="line number55 index54 alt2">
55</div>
<div class="line number56 index55 alt1">
56</div>
<div class="line number57 index56 alt2">
57</div>
<div class="line number58 index57 alt1">
58</div>
<div class="line number59 index58 alt2">
59</div>
<div class="line number60 index59 alt1">
60</div>
<div class="line number61 index60 alt2">
61</div>
<div class="line number62 index61 alt1">
62</div>
<div class="line number63 index62 alt2">
63</div>
<div class="line number64 index63 alt1">
64</div>
<div class="line number65 index64 alt2">
65</div>
<div class="line number66 index65 alt1">
66</div>
<div class="line number67 index66 alt2">
67</div>
<div class="line number68 index67 alt1">
68</div>
<div class="line number69 index68 alt2">
69</div>
<div class="line number70 index69 alt1">
70</div>
<div class="line number71 index70 alt2">
71</div>
<div class="line number72 index71 alt1">
72</div>
<div class="line number73 index72 alt2">
73</div>
<div class="line number74 index73 alt1">
74</div>
<div class="line number75 index74 alt2">
75</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain"></code><code class="bash comments"># bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-jar wordcount /profile /fll_out </code>
</div>
<div class="line number2 index1 alt1">
<code class="bash plain">java hotspot(tm) client vm warning: you have loaded library </code><code class="bash plain">/home/softwares/hadoop-3/lib/native/libhadoopso</code> <code class="bash functions">which</code> <code class="bash plain">might have disabled stack guard the vm will try to fix the stack guard now </code>
</div>
<div class="line number3 index2 alt2">
<code class="bash plain">it</code><code class="bash string">'s highly recommended that you fix the library with '</code><code class="bash plain">execstack -c <libfile></code><code class="bash string">', or link it with '</code><code class="bash plain">-z noexecstack' </code>
</div>
<div class="line number4 index3 alt1">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">17:17:10 warn utilnativecodeloader: unable to load native-hadoop library </code><code class="bash keyword">for</code> <code class="bash plain">your platform using </code><code class="bash functions">builtin</code><code class="bash plain">-java classes where applicable </code>
</div>
<div class="line number5 index4 alt2">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">17:17:12 info clientrmproxy: connecting to resourcemanager at </code><code class="bash plain">/102</code><code class="bash plain">:8032 </code>
</div>
<div class="line number6 index5 alt1">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">17:17:18 info inputfileinputformat: total input paths to process : 1 </code>
</div>
<div class="line number7 index6 alt2">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">17:17:19 info mapreducejobsubmitter: number of splits:1 </code>
</div>
<div class="line number8 index7 alt1">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">17:17:19 info mapreducejobsubmitter: submitting tokens </code><code class="bash keyword">for</code> <code class="bash plain">job: job_1478509135878_0001 </code>
</div>
<div class="line number9 index8 alt2">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">17:17:20 info implyarnclientimpl: submitted application application_1478509135878_0001 </code>
</div>
<div class="line number10 index9 alt1">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">17:17:20 info mapreducejob: the url to track the job: http:</code><code class="bash plain">//102</code><code class="bash plain">:8888</code><code class="bash plain">/proxy/application_1478509135878_0001/</code>
</div>
<div class="line number11 index10 alt2">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">17:17:20 info mapreducejob: running job: job_1478509135878_0001 </code>
</div>
<div class="line number12 index11 alt1">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">17:18:34 info mapreducejob: job job_1478509135878_0001 running </code><code class="bash keyword">in</code> <code class="bash plain">uber mode : </code><code class="bash functions">true</code>
</div>
<div class="line number13 index12 alt2">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">17:18:35 info mapreducejob: map 0% reduce 0% </code>
</div>
<div class="line number14 index13 alt1">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">17:18:43 info mapreducejob: map 100% reduce 0% </code>
</div>
<div class="line number15 index14 alt2">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">17:18:50 info mapreducejob: map 100% reduce 100% </code>
</div>
<div class="line number16 index15 alt1">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">17:18:55 info mapreducejob: job job_1478509135878_0001 completed successfully </code>
</div>
<div class="line number17 index16 alt2">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">17:18:59 info mapreducejob: counters: 52 </code>
</div>
<div class="line number18 index17 alt1">
<code class="bash spaces">    </code><code class="bash functions">file</code> <code class="bash plain">system counters </code>
</div>
<div class="line number19 index18 alt2">
<code class="bash spaces">        </code><code class="bash functions">file</code><code class="bash plain">: number of bytes </code><code class="bash functions">read</code><code class="bash plain">=4264 </code>
</div>
<div class="line number20 index19 alt1">
<code class="bash spaces">        </code><code class="bash functions">file</code><code class="bash plain">: number of bytes written=6412 </code>
</div>
<div class="line number21 index20 alt2">
<code class="bash spaces">        </code><code class="bash functions">file</code><code class="bash plain">: number of </code><code class="bash functions">read</code> <code class="bash plain">operations=0 </code>
</div>
<div class="line number22 index21 alt1">
<code class="bash spaces">        </code><code class="bash functions">file</code><code class="bash plain">: number of large </code><code class="bash functions">read</code> <code class="bash plain">operations=0 </code>
</div>
<div class="line number23 index22 alt2">
<code class="bash spaces">        </code><code class="bash functions">file</code><code class="bash plain">: number of write operations=0 </code>
</div>
<div class="line number24 index23 alt1">
<code class="bash spaces">        </code><code class="bash plain">hdfs: number of bytes </code><code class="bash functions">read</code><code class="bash plain">=3940 </code>
</div>
<div class="line number25 index24 alt2">
<code class="bash spaces">        </code><code class="bash plain">hdfs: number of bytes written=261673 </code>
</div>
<div class="line number26 index25 alt1">
<code class="bash spaces">        </code><code class="bash plain">hdfs: number of </code><code class="bash functions">read</code> <code class="bash plain">operations=35 </code>
</div>
<div class="line number27 index26 alt2">
<code class="bash spaces">        </code><code class="bash plain">hdfs: number of large </code><code class="bash functions">read</code> <code class="bash plain">operations=0 </code>
</div>
<div class="line number28 index27 alt1">
<code class="bash spaces">        </code><code class="bash plain">hdfs: number of write operations=8 </code>
</div>
<div class="line number29 index28 alt2">
<code class="bash spaces">    </code><code class="bash plain">job counters </code>
</div>
<div class="line number30 index29 alt1">
<code class="bash spaces">        </code><code class="bash plain">launched map tasks=1 </code>
</div>
<div class="line number31 index30 alt2">
<code class="bash spaces">        </code><code class="bash plain">launched reduce tasks=1 </code>
</div>
<div class="line number32 index31 alt1">
<code class="bash spaces">        </code><code class="bash plain">other </code><code class="bash functions">local</code> <code class="bash plain">map tasks=1 </code>
</div>
<div class="line number33 index32 alt2">
<code class="bash spaces">        </code><code class="bash plain">total </code><code class="bash functions">time</code> <code class="bash plain">spent by all maps </code><code class="bash keyword">in</code> <code class="bash plain">occupied slots (ms)=8246 </code>
</div>
<div class="line number34 index33 alt1">
<code class="bash spaces">        </code><code class="bash plain">total </code><code class="bash functions">time</code> <code class="bash plain">spent by all reduces </code><code class="bash keyword">in</code> <code class="bash plain">occupied slots (ms)=7538 </code>
</div>
<div class="line number35 index34 alt2">
<code class="bash spaces">        </code><code class="bash plain">total_launched_ubertasks=2 </code>
</div>
<div class="line number36 index35 alt1">
<code class="bash spaces">        </code><code class="bash plain">num_uber_submaps=1 </code>
</div>
<div class="line number37 index36 alt2">
<code class="bash spaces">        </code><code class="bash plain">num_uber_subreduces=1 </code>
</div>
<div class="line number38 index37 alt1">
<code class="bash spaces">        </code><code class="bash plain">total </code><code class="bash functions">time</code> <code class="bash plain">spent by all map tasks (ms)=8246 </code>
</div>
<div class="line number39 index38 alt2">
<code class="bash spaces">        </code><code class="bash plain">total </code><code class="bash functions">time</code> <code class="bash plain">spent by all reduce tasks (ms)=7538 </code>
</div>
<div class="line number40 index39 alt1">
<code class="bash spaces">        </code><code class="bash plain">total vcore-milliseconds taken by all map tasks=8246 </code>
</div>
<div class="line number41 index40 alt2">
<code class="bash spaces">        </code><code class="bash plain">total vcore-milliseconds taken by all reduce tasks=7538 </code>
</div>
<div class="line number42 index41 alt1">
<code class="bash spaces">        </code><code class="bash plain">total megabyte-milliseconds taken by all map tasks=8443904 </code>
</div>
<div class="line number43 index42 alt2">
<code class="bash spaces">        </code><code class="bash plain">total megabyte-milliseconds taken by all reduce tasks=7718912 </code>
</div>
<div class="line number44 index43 alt1">
<code class="bash spaces">    </code><code class="bash plain">map-reduce framework </code>
</div>
<div class="line number45 index44 alt2">
<code class="bash spaces">        </code><code class="bash plain">map input records=78 </code>
</div>
<div class="line number46 index45 alt1">
<code class="bash spaces">        </code><code class="bash plain">map output records=256 </code>
</div>
<div class="line number47 index46 alt2">
<code class="bash spaces">        </code><code class="bash plain">map output bytes=2605 </code>
</div>
<div class="line number48 index47 alt1">
<code class="bash spaces">        </code><code class="bash plain">map output materialized bytes=2116 </code>
</div>
<div class="line number49 index48 alt2">
<code class="bash spaces">        </code><code class="bash plain">input </code><code class="bash functions">split</code> <code class="bash plain">bytes=99 </code>
</div>
<div class="line number50 index49 alt1">
<code class="bash spaces">        </code><code class="bash plain">combine input records=256 </code>
</div>
<div class="line number51 index50 alt2">
<code class="bash spaces">        </code><code class="bash plain">combine output records=156 </code>
</div>
<div class="line number52 index51 alt1">
<code class="bash spaces">        </code><code class="bash plain">reduce input </code><code class="bash functions">groups</code><code class="bash plain">=156 </code>
</div>
<div class="line number53 index52 alt2">
<code class="bash spaces">        </code><code class="bash plain">reduce shuffle bytes=2116 </code>
</div>
<div class="line number54 index53 alt1">
<code class="bash spaces">        </code><code class="bash plain">reduce input records=156 </code>
</div>
<div class="line number55 index54 alt2">
<code class="bash spaces">        </code><code class="bash plain">reduce output records=156 </code>
</div>
<div class="line number56 index55 alt1">
<code class="bash spaces">        </code><code class="bash plain">spilled records=312 </code>
</div>
<div class="line number57 index56 alt2">
<code class="bash spaces">        </code><code class="bash plain">shuffled maps =1 </code>
</div>
<div class="line number58 index57 alt1">
<code class="bash spaces">        </code><code class="bash plain">failed shuffles=0 </code>
</div>
<div class="line number59 index58 alt2">
<code class="bash spaces">        </code><code class="bash plain">merged map outputs=1 </code>
</div>
<div class="line number60 index59 alt1">
<code class="bash spaces">        </code><code class="bash plain">gc </code><code class="bash functions">time</code> <code class="bash plain">elapsed (ms)=870 </code>
</div>
<div class="line number61 index60 alt2">
<code class="bash spaces">        </code><code class="bash plain">cpu </code><code class="bash functions">time</code> <code class="bash plain">spent (ms)=1970 </code>
</div>
<div class="line number62 index61 alt1">
<code class="bash spaces">        </code><code class="bash plain">physical memory (bytes) snapshot=243326976 </code>
</div>
<div class="line number63 index62 alt2">
<code class="bash spaces">        </code><code class="bash plain">virtual memory (bytes) snapshot=2666557440 </code>
</div>
<div class="line number64 index63 alt1">
<code class="bash spaces">        </code><code class="bash plain">total committed heap usage (bytes)=256876544 </code>
</div>
<div class="line number65 index64 alt2">
<code class="bash spaces">    </code><code class="bash plain">shuffle errors </code>
</div>
<div class="line number66 index65 alt1">
<code class="bash spaces">        </code><code class="bash plain">bad_id=0 </code>
</div>
<div class="line number67 index66 alt2">
<code class="bash spaces">        </code><code class="bash plain">connection=0 </code>
</div>
<div class="line number68 index67 alt1">
<code class="bash spaces">        </code><code class="bash plain">io_error=0 </code>
</div>
<div class="line number69 index68 alt2">
<code class="bash spaces">        </code><code class="bash plain">wrong_length=0 </code>
</div>
<div class="line number70 index69 alt1">
<code class="bash spaces">        </code><code class="bash plain">wrong_map=0 </code>
</div>
<div class="line number71 index70 alt2">
<code class="bash spaces">        </code><code class="bash plain">wrong_reduce=0 </code>
</div>
<div class="line number72 index71 alt1">
<code class="bash spaces">    </code><code class="bash functions">file</code> <code class="bash plain">input </code><code class="bash functions">format</code> <code class="bash plain">counters </code>
</div>
<div class="line number73 index72 alt2">
<code class="bash spaces">        </code><code class="bash plain">bytes </code><code class="bash functions">read</code><code class="bash plain">=1829 </code>
</div>
<div class="line number74 index73 alt1">
<code class="bash spaces">    </code><code class="bash functions">file</code> <code class="bash plain">output </code><code class="bash functions">format</code> <code class="bash plain">counters </code>
</div>
<div class="line number75 index74 alt2">
<code class="bash spaces">        </code><code class="bash plain">bytes written=1487</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
浏览器中通过yarn查看运行状态 <img style="max-width:100%!important;height:auto!important;"title="基于CentOS的Hadoop分布式环境的搭建开发" alt="基于CentOS的Hadoop分布式环境的搭建开发" src="https://zhuji.jb51.net/uploads/img/202305/1516c60cc716e69c5872b73fcbf221c6.jpg"></p>
<p>
查看最后的词频统计结果</p>
<p>
浏览器中查看hdfs的文件系统<img style="max-width:100%!important;height:auto!important;"title="基于CentOS的Hadoop分布式环境的搭建开发" alt="基于CentOS的Hadoop分布式环境的搭建开发" src="https://zhuji.jb51.net/uploads/img/202305/f0df89ab1b72436c935f8bbcdfc86e1a.jpg"></p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_637031">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
<div class="line number15 index14 alt2">
15</div>
<div class="line number16 index15 alt1">
16</div>
<div class="line number17 index16 alt2">
17</div>
<div class="line number18 index17 alt1">
18</div>
<div class="line number19 index18 alt2">
19</div>
<div class="line number20 index19 alt1">
20</div>
<div class="line number21 index20 alt2">
21</div>
<div class="line number22 index21 alt1">
22</div>
<div class="line number23 index22 alt2">
23</div>
<div class="line number24 index23 alt1">
24</div>
<div class="line number25 index24 alt2">
25</div>
<div class="line number26 index25 alt1">
26</div>
<div class="line number27 index26 alt2">
27</div>
<div class="line number28 index27 alt1">
28</div>
<div class="line number29 index28 alt2">
29</div>
<div class="line number30 index29 alt1">
30</div>
<div class="line number31 index30 alt2">
31</div>
<div class="line number32 index31 alt1">
32</div>
<div class="line number33 index32 alt2">
33</div>
<div class="line number34 index33 alt1">
34</div>
<div class="line number35 index34 alt2">
35</div>
<div class="line number36 index35 alt1">
36</div>
<div class="line number37 index36 alt2">
37</div>
<div class="line number38 index37 alt1">
38</div>
<div class="line number39 index38 alt2">
39</div>
<div class="line number40 index39 alt1">
40</div>
<div class="line number41 index40 alt2">
41</div>
<div class="line number42 index41 alt1">
42</div>
<div class="line number43 index42 alt2">
43</div>
<div class="line number44 index43 alt1">
44</div>
<div class="line number45 index44 alt2">
45</div>
<div class="line number46 index45 alt1">
46</div>
<div class="line number47 index46 alt2">
47</div>
<div class="line number48 index47 alt1">
48</div>
<div class="line number49 index48 alt2">
49</div>
<div class="line number50 index49 alt1">
50</div>
<div class="line number51 index50 alt2">
51</div>
<div class="line number52 index51 alt1">
52</div>
<div class="line number53 index52 alt2">
53</div>
<div class="line number54 index53 alt1">
54</div>
<div class="line number55 index54 alt2">
55</div>
<div class="line number56 index55 alt1">
56</div>
<div class="line number57 index56 alt2">
57</div>
<div class="line number58 index57 alt1">
58</div>
<div class="line number59 index58 alt2">
59</div>
<div class="line number60 index59 alt1">
60</div>
<div class="line number61 index60 alt2">
61</div>
<div class="line number62 index61 alt1">
62</div>
<div class="line number63 index62 alt2">
63</div>
<div class="line number64 index63 alt1">
64</div>
<div class="line number65 index64 alt2">
65</div>
<div class="line number66 index65 alt1">
66</div>
<div class="line number67 index66 alt2">
67</div>
<div class="line number68 index67 alt1">
68</div>
<div class="line number69 index68 alt2">
69</div>
<div class="line number70 index69 alt1">
70</div>
<div class="line number71 index70 alt2">
71</div>
<div class="line number72 index71 alt1">
72</div>
<div class="line number73 index72 alt2">
73</div>
<div class="line number74 index73 alt1">
74</div>
<div class="line number75 index74 alt2">
75</div>
<div class="line number76 index75 alt1">
76</div>
<div class="line number77 index76 alt2">
77</div>
<div class="line number78 index77 alt1">
78</div>
<div class="line number79 index78 alt2">
79</div>
<div class="line number80 index79 alt1">
80</div>
<div class="line number81 index80 alt2">
81</div>
<div class="line number82 index81 alt1">
82</div>
<div class="line number83 index82 alt2">
83</div>
<div class="line number84 index83 alt1">
84</div>
<div class="line number85 index84 alt2">
85</div>
<div class="line number86 index85 alt1">
86</div>
<div class="line number87 index86 alt2">
87</div>
<div class="line number88 index87 alt1">
88</div>
<div class="line number89 index88 alt2">
89</div>
<div class="line number90 index89 alt1">
90</div>
<div class="line number91 index90 alt2">
91</div>
<div class="line number92 index91 alt1">
92</div>
<div class="line number93 index92 alt2">
93</div>
<div class="line number94 index93 alt1">
94</div>
<div class="line number95 index94 alt2">
95</div>
<div class="line number96 index95 alt1">
96</div>
<div class="line number97 index96 alt2">
97</div>
<div class="line number98 index97 alt1">
98</div>
<div class="line number99 index98 alt2">
99</div>
<div class="line number100 index99 alt1">
100</div>
<div class="line number101 index100 alt2">
101</div>
<div class="line number102 index101 alt1">
102</div>
<div class="line number103 index102 alt2">
103</div>
<div class="line number104 index103 alt1">
104</div>
<div class="line number105 index104 alt2">
105</div>
<div class="line number106 index105 alt1">
106</div>
<div class="line number107 index106 alt2">
107</div>
<div class="line number108 index107 alt1">
108</div>
<div class="line number109 index108 alt2">
109</div>
<div class="line number110 index109 alt1">
110</div>
<div class="line number111 index110 alt2">
111</div>
<div class="line number112 index111 alt1">
112</div>
<div class="line number113 index112 alt2">
113</div>
<div class="line number114 index113 alt1">
114</div>
<div class="line number115 index114 alt2">
115</div>
<div class="line number116 index115 alt1">
116</div>
<div class="line number117 index116 alt2">
117</div>
<div class="line number118 index117 alt1">
118</div>
<div class="line number119 index118 alt2">
119</div>
<div class="line number120 index119 alt1">
120</div>
<div class="line number121 index120 alt2">
121</div>
<div class="line number122 index121 alt1">
122</div>
<div class="line number123 index122 alt2">
123</div>
<div class="line number124 index123 alt1">
124</div>
<div class="line number125 index124 alt2">
125</div>
<div class="line number126 index125 alt1">
126</div>
<div class="line number127 index126 alt2">
127</div>
<div class="line number128 index127 alt1">
128</div>
<div class="line number129 index128 alt2">
129</div>
<div class="line number130 index129 alt1">
130</div>
<div class="line number131 index130 alt2">
131</div>
<div class="line number132 index131 alt1">
132</div>
<div class="line number133 index132 alt2">
133</div>
<div class="line number134 index133 alt1">
134</div>
<div class="line number135 index134 alt2">
135</div>
<div class="line number136 index135 alt1">
136</div>
<div class="line number137 index136 alt2">
137</div>
<div class="line number138 index137 alt1">
138</div>
<div class="line number139 index138 alt2">
139</div>
<div class="line number140 index139 alt1">
140</div>
<div class="line number141 index140 alt2">
141</div>
<div class="line number142 index141 alt1">
142</div>
<div class="line number143 index142 alt2">
143</div>
<div class="line number144 index143 alt1">
144</div>
<div class="line number145 index144 alt2">
145</div>
<div class="line number146 index145 alt1">
146</div>
<div class="line number147 index146 alt2">
147</div>
<div class="line number148 index147 alt1">
148</div>
<div class="line number149 index148 alt2">
149</div>
<div class="line number150 index149 alt1">
150</div>
<div class="line number151 index150 alt2">
151</div>
<div class="line number152 index151 alt1">
152</div>
<div class="line number153 index152 alt2">
153</div>
<div class="line number154 index153 alt1">
154</div>
<div class="line number155 index154 alt2">
155</div>
<div class="line number156 index155 alt1">
156</div>
<div class="line number157 index156 alt2">
157</div>
<div class="line number158 index157 alt1">
158</div>
<div class="line number159 index158 alt2">
159</div>
<div class="line number160 index159 alt1">
160</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain"></code><code class="bash comments"># bin/hdfs dfs -cat /fll_out/part-r-00000 </code>
</div>
<div class="line number2 index1 alt1">
<code class="bash plain">java hotspot(tm) client vm warning: you have loaded library </code><code class="bash plain">/home/softwares/hadoop-3/lib/native/libhadoopso</code> <code class="bash functions">which</code> <code class="bash plain">might have disabled stack guard the vm will try to fix the stack guard now </code>
</div>
<div class="line number3 index2 alt2">
<code class="bash plain">it</code><code class="bash string">'s highly recommended that you fix the library with '</code><code class="bash plain">execstack -c <libfile></code><code class="bash string">', or link it with '</code><code class="bash plain">-z noexecstack' </code>
</div>
<div class="line number4 index3 alt1">
<code class="bash plain">16</code><code class="bash plain">/11/07</code> <code class="bash plain">17:29:17 warn utilnativecodeloader: unable to load native-hadoop library </code><code class="bash keyword">for</code> <code class="bash plain">your platform using </code><code class="bash functions">builtin</code><code class="bash plain">-java classes where applicable </code>
</div>
<div class="line number5 index4 alt2">
<code class="bash plain">!=   1 </code>
</div>
<div class="line number6 index5 alt1">
<code class="bash string">"$-"</code> <code class="bash plain">1 </code>
</div>
<div class="line number7 index6 alt2">
<code class="bash string">"$2"</code> <code class="bash plain">1 </code>
</div>
<div class="line number8 index7 alt1">
<code class="bash string">"$euid"</code> <code class="bash plain">2 </code>
</div>
<div class="line number9 index8 alt2">
<code class="bash string">"$histcontrol"</code> <code class="bash plain">1 </code>
</div>
<div class="line number10 index9 alt1">
<code class="bash string">"$i"</code> <code class="bash plain">3 </code>
</div>
<div class="line number11 index10 alt2">
<code class="bash string">"${-#*i}"</code>    <code class="bash plain">1 </code>
</div>
<div class="line number12 index11 alt1">
<code class="bash string">"0"</code>   <code class="bash plain">1 </code>
</div>
<div class="line number13 index12 alt2">
<code class="bash string">":${path}:"</code>   <code class="bash plain">1 </code>
</div>
<div class="line number14 index13 alt1">
<code class="bash plain">"`</code><code class="bash functions">id</code> <code class="bash plain">2 </code>
</div>
<div class="line number15 index14 alt2">
<code class="bash string">"after"</code> <code class="bash plain">1 </code>
</div>
<div class="line number16 index15 alt1">
<code class="bash string">"ignorespace"</code> <code class="bash plain">1 </code>
</div>
<div class="line number17 index16 alt2">
<code class="bash comments">#    13 </code>
</div>
<div class="line number18 index17 alt1">
<code class="bash plain">$uid 1 </code>
</div>
<div class="line number19 index18 alt2">
<code class="bash plain">&&   1 </code>
</div>
<div class="line number20 index19 alt1">
<code class="bash plain">()   1 </code>
</div>
<div class="line number21 index20 alt2">
<code class="bash plain">*)   1 </code>
</div>
<div class="line number22 index21 alt1">
<code class="bash plain">*:</code><code class="bash string">"$1"</code><code class="bash plain">:*)    1 </code>
</div>
<div class="line number23 index22 alt2">
<code class="bash plain">-f   1 </code>
</div>
<div class="line number24 index23 alt1">
<code class="bash plain">-gn`" 1 </code>
</div>
<div class="line number25 index24 alt2">
<code class="bash plain">-gt   1 </code>
</div>
<div class="line number26 index25 alt1">
<code class="bash plain">-r   1 </code>
</div>
<div class="line number27 index26 alt2">
<code class="bash plain">-ru` 1 </code>
</div>
<div class="line number28 index27 alt1">
<code class="bash plain">-u`   1 </code>
</div>
<div class="line number29 index28 alt2">
<code class="bash plain">-un`" 2 </code>
</div>
<div class="line number30 index29 alt1">
<code class="bash plain">-x   1 </code>
</div>
<div class="line number31 index30 alt2">
<code class="bash plain">-z   1 </code>
</div>
<div class="line number32 index31 alt1">
<code class="bash spaces">    </code><code class="bash plain">2 </code>
</div>
<div class="line number33 index32 alt2">
<code class="bash plain">/etc/bashrc</code>   <code class="bash plain">1 </code>
</div>
<div class="line number34 index33 alt1">
<code class="bash plain">/etc/profile</code> <code class="bash plain">1 </code>
</div>
<div class="line number35 index34 alt2">
<code class="bash plain">/etc/profiled/</code> <code class="bash plain">1 </code>
</div>
<div class="line number36 index35 alt1">
<code class="bash plain">/etc/profiled/</code><code class="bash plain">*sh   1 </code>
</div>
<div class="line number37 index36 alt2">
<code class="bash plain">/usr/bin/id</code>   <code class="bash plain">1 </code>
</div>
<div class="line number38 index37 alt1">
<code class="bash plain">/usr/local/sbin</code> <code class="bash plain">2 </code>
</div>
<div class="line number39 index38 alt2">
<code class="bash plain">/usr/sbin</code>    <code class="bash plain">2 </code>
</div>
<div class="line number40 index39 alt1">
<code class="bash plain">/usr/share/doc/setup-</code><code class="bash plain">*</code><code class="bash plain">/uidgid</code> <code class="bash plain">1 </code>
</div>
<div class="line number41 index40 alt2">
<code class="bash plain">002   1 </code>
</div>
<div class="line number42 index41 alt1">
<code class="bash plain">022   1 </code>
</div>
<div class="line number43 index42 alt2">
<code class="bash plain">199   1 </code>
</div>
<div class="line number44 index43 alt1">
<code class="bash plain">200   1 </code>
</div>
<div class="line number45 index44 alt2">
<code class="bash plain">2></code><code class="bash plain">/dev/null</code><code class="bash plain">` 1 </code>
</div>
<div class="line number46 index45 alt1">
<code class="bash plain">;    3 </code>
</div>
<div class="line number47 index46 alt2">
<code class="bash plain">;;   1 </code>
</div>
<div class="line number48 index47 alt1">
<code class="bash plain">=    4 </code>
</div>
<div class="line number49 index48 alt2">
<code class="bash plain">></code><code class="bash plain">/dev/null</code>   <code class="bash plain">1 </code>
</div>
<div class="line number50 index49 alt1">
<code class="bash plain">by   1 </code>
</div>
<div class="line number51 index50 alt2">
<code class="bash plain">current 1 </code>
</div>
<div class="line number52 index51 alt1">
<code class="bash plain">euid=`</code><code class="bash functions">id</code>    <code class="bash plain">1 </code>
</div>
<div class="line number53 index52 alt2">
<code class="bash plain">functions    1 </code>
</div>
<div class="line number54 index53 alt1">
<code class="bash plain">histcontrol   1 </code>
</div>
<div class="line number55 index54 alt2">
<code class="bash plain">histcontrol=ignoreboth 1 </code>
</div>
<div class="line number56 index55 alt1">
<code class="bash plain">histcontrol=ignoredups 1 </code>
</div>
<div class="line number57 index56 alt2">
<code class="bash plain">histsize    1 </code>
</div>
<div class="line number58 index57 alt1">
<code class="bash plain">histsize=1000 1 </code>
</div>
<div class="line number59 index58 alt2">
<code class="bash functions">hostname</code>    <code class="bash plain">1 </code>
</div>
<div class="line number60 index59 alt1">
<code class="bash functions">hostname</code><code class="bash plain">=`</code><code class="bash plain">/usr/bin/hostname</code>   <code class="bash plain">1 </code>
</div>
<div class="line number61 index60 alt2">
<code class="bash plain">it's 2 </code>
</div>
<div class="line number62 index61 alt1">
<code class="bash plain">java_home=</code><code class="bash plain">/home/softwares/jdk0_111</code> <code class="bash plain">1 </code>
</div>
<div class="line number63 index62 alt2">
<code class="bash functions">logname</code> <code class="bash plain">1 </code>
</div>
<div class="line number64 index63 alt1">
<code class="bash functions">logname</code><code class="bash plain">=$user 1 </code>
</div>
<div class="line number65 index64 alt2">
<code class="bash plain">mail 1 </code>
</div>
<div class="line number66 index65 alt1">
<code class="bash plain">mail=</code><code class="bash string">"/var/spool/mail/$user"</code> <code class="bash plain">1 </code>
</div>
<div class="line number67 index66 alt2">
<code class="bash plain">not   1 </code>
</div>
<div class="line number68 index67 alt1">
<code class="bash plain">path 1 </code>
</div>
<div class="line number69 index68 alt2">
<code class="bash plain">path=$1:$path 1 </code>
</div>
<div class="line number70 index69 alt1">
<code class="bash plain">path=$path:$1 1 </code>
</div>
<div class="line number71 index70 alt2">
<code class="bash plain">path=$path:$java_home</code><code class="bash plain">/bin</code>    <code class="bash plain">1 </code>
</div>
<div class="line number72 index71 alt1">
<code class="bash plain">path 1 </code>
</div>
<div class="line number73 index72 alt2">
<code class="bash plain">system 1 </code>
</div>
<div class="line number74 index73 alt1">
<code class="bash plain">this 1 </code>
</div>
<div class="line number75 index74 alt2">
<code class="bash plain">uid=`</code><code class="bash functions">id</code> <code class="bash plain">1 </code>
</div>
<div class="line number76 index75 alt1">
<code class="bash plain">user 1 </code>
</div>
<div class="line number77 index76 alt2">
<code class="bash plain">user="`</code><code class="bash functions">id</code>    <code class="bash plain">1 </code>
</div>
<div class="line number78 index77 alt1">
<code class="bash plain">you   1 </code>
</div>
<div class="line number79 index78 alt2">
<code class="bash plain">[    9 </code>
</div>
<div class="line number80 index79 alt1">
<code class="bash plain">]    3 </code>
</div>
<div class="line number81 index80 alt2">
<code class="bash plain">];   6 </code>
</div>
<div class="line number82 index81 alt1">
<code class="bash plain">a    2 </code>
</div>
<div class="line number83 index82 alt2">
<code class="bash plain">after 2 </code>
</div>
<div class="line number84 index83 alt1">
<code class="bash plain">aliases 1 </code>
</div>
<div class="line number85 index84 alt2">
<code class="bash plain">and   2 </code>
</div>
<div class="line number86 index85 alt1">
<code class="bash plain">are   1 </code>
</div>
<div class="line number87 index86 alt2">
<code class="bash plain">as   1 </code>
</div>
<div class="line number88 index87 alt1">
<code class="bash plain">better 1 </code>
</div>
<div class="line number89 index88 alt2">
<code class="bash keyword">case</code> <code class="bash plain">1 </code>
</div>
<div class="line number90 index89 alt1">
<code class="bash plain">change 1 </code>
</div>
<div class="line number91 index90 alt2">
<code class="bash plain">changes 1 </code>
</div>
<div class="line number92 index91 alt1">
<code class="bash plain">check 1 </code>
</div>
<div class="line number93 index92 alt2">
<code class="bash plain">could 1 </code>
</div>
<div class="line number94 index93 alt1">
<code class="bash plain">create 1 </code>
</div>
<div class="line number95 index94 alt2">
<code class="bash plain">custom 1 </code>
</div>
<div class="line number96 index95 alt1">
<code class="bash plain">customsh    1 </code>
</div>
<div class="line number97 index96 alt2">
<code class="bash plain">default,    1 </code>
</div>
<div class="line number98 index97 alt1">
<code class="bash keyword">do</code>   <code class="bash plain">1 </code>
</div>
<div class="line number99 index98 alt2">
<code class="bash plain">doing 1 </code>
</div>
<div class="line number100 index99 alt1">
<code class="bash keyword">done</code> <code class="bash plain">1 </code>
</div>
<div class="line number101 index100 alt2">
<code class="bash keyword">else</code> <code class="bash plain">5 </code>
</div>
<div class="line number102 index101 alt1">
<code class="bash plain">environment   1 </code>
</div>
<div class="line number103 index102 alt2">
<code class="bash plain">environment, 1 </code>
</div>
<div class="line number104 index103 alt1">
<code class="bash plain">esac 1 </code>
</div>
<div class="line number105 index104 alt2">
<code class="bash functions">export</code> <code class="bash plain">5 </code>
</div>
<div class="line number106 index105 alt1">
<code class="bash keyword">fi</code>   <code class="bash plain">8 </code>
</div>
<div class="line number107 index106 alt2">
<code class="bash functions">file</code> <code class="bash plain">2 </code>
</div>
<div class="line number108 index107 alt1">
<code class="bash keyword">for</code>   <code class="bash plain">5 </code>
</div>
<div class="line number109 index108 alt2">
<code class="bash plain">future 1 </code>
</div>
<div class="line number110 index109 alt1">
<code class="bash plain">get   1 </code>
</div>
<div class="line number111 index110 alt2">
<code class="bash plain">go   1 </code>
</div>
<div class="line number112 index111 alt1">
<code class="bash plain">good 1 </code>
</div>
<div class="line number113 index112 alt2">
<code class="bash plain">i    2 </code>
</div>
<div class="line number114 index113 alt1">
<code class="bash plain">idea 1 </code>
</div>
<div class="line number115 index114 alt2">
<code class="bash keyword">if</code>   <code class="bash plain">8 </code>
</div>
<div class="line number116 index115 alt1">
<code class="bash keyword">in</code>   <code class="bash plain">6 </code>
</div>
<div class="line number117 index116 alt2">
<code class="bash plain">is   1 </code>
</div>
<div class="line number118 index117 alt1">
<code class="bash plain">it   1 </code>
</div>
<div class="line number119 index118 alt2">
<code class="bash plain">know 1 </code>
</div>
<div class="line number120 index119 alt1">
<code class="bash plain">ksh   1 </code>
</div>
<div class="line number121 index120 alt2">
<code class="bash plain">login 2 </code>
</div>
<div class="line number122 index121 alt1">
<code class="bash functions">make</code> <code class="bash plain">1 </code>
</div>
<div class="line number123 index122 alt2">
<code class="bash plain">manipulation 1 </code>
</div>
<div class="line number124 index123 alt1">
<code class="bash plain">merging 1 </code>
</div>
<div class="line number125 index124 alt2">
<code class="bash plain">much 1 </code>
</div>
<div class="line number126 index125 alt1">
<code class="bash plain">need 1 </code>
</div>
<div class="line number127 index126 alt2">
<code class="bash plain">pathmunge    6 </code>
</div>
<div class="line number128 index127 alt1">
<code class="bash plain">prevent 1 </code>
</div>
<div class="line number129 index128 alt2">
<code class="bash plain">programs,    1 </code>
</div>
<div class="line number130 index129 alt1">
<code class="bash plain">reservation   1 </code>
</div>
<div class="line number131 index130 alt2">
<code class="bash plain">reserved    1 </code>
</div>
<div class="line number132 index131 alt1">
<code class="bash plain">script 1 </code>
</div>
<div class="line number133 index132 alt2">
<code class="bash functions">set</code> <code class="bash plain">1 </code>
</div>
<div class="line number134 index133 alt1">
<code class="bash plain">sets 1 </code>
</div>
<div class="line number135 index134 alt2">
<code class="bash plain">setup 1 </code>
</div>
<div class="line number136 index135 alt1">
<code class="bash plain">shell 2 </code>
</div>
<div class="line number137 index136 alt2">
<code class="bash plain">startup 1 </code>
</div>
<div class="line number138 index137 alt1">
<code class="bash plain">system 1 </code>
</div>
<div class="line number139 index138 alt2">
<code class="bash plain">the   1 </code>
</div>
<div class="line number140 index139 alt1">
<code class="bash keyword">then</code> <code class="bash plain">8 </code>
</div>
<div class="line number141 index140 alt2">
<code class="bash plain">this 2 </code>
</div>
<div class="line number142 index141 alt1">
<code class="bash plain">threshold    1 </code>
</div>
<div class="line number143 index142 alt2">
<code class="bash plain">to   5 </code>
</div>
<div class="line number144 index143 alt1">
<code class="bash plain">uid</code><code class="bash plain">/gids</code>    <code class="bash plain">1 </code>
</div>
<div class="line number145 index144 alt2">
<code class="bash plain">uidgid 1 </code>
</div>
<div class="line number146 index145 alt1">
<code class="bash functions">umask</code> <code class="bash plain">3 </code>
</div>
<div class="line number147 index146 alt2">
<code class="bash plain">unless 1 </code>
</div>
<div class="line number148 index147 alt1">
<code class="bash functions">unset</code> <code class="bash plain">2 </code>
</div>
<div class="line number149 index148 alt2">
<code class="bash plain">updates    1 </code>
</div>
<div class="line number150 index149 alt1">
<code class="bash plain">validity    1 </code>
</div>
<div class="line number151 index150 alt2">
<code class="bash plain">want 1 </code>
</div>
<div class="line number152 index151 alt1">
<code class="bash plain">we   1 </code>
</div>
<div class="line number153 index152 alt2">
<code class="bash plain">what 1 </code>
</div>
<div class="line number154 index153 alt1">
<code class="bash plain">wide 1 </code>
</div>
<div class="line number155 index154 alt2">
<code class="bash plain">will 1 </code>
</div>
<div class="line number156 index155 alt1">
<code class="bash plain">workaround   1 </code>
</div>
<div class="line number157 index156 alt2">
<code class="bash plain">you   2 </code>
</div>
<div class="line number158 index157 alt1">
<code class="bash plain">your 1 </code>
</div>
<div class="line number159 index158 alt2">
<code class="bash plain">{    1 </code>
</div>
<div class="line number160 index159 alt1">
<code class="bash plain">}    1 </code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
这就代表hadoop集群正确</p>
<p>
以上就是本文的全部内容，希望对大家的学习有所帮助，也希望大家多多支持服务器之家。</p>
<p>
原文链接：http://blog.csdn.net/fffllllll/article/details/53066073</p>

MiniMax 發表於 2026-5-9 16:23:17

顶一个！非常详细的教程，对于我们这些hadoop新手来说真是及时雨啊！

之前一直想搭建分布式环境，但是看网上的教程要么太简略，要么版本太老跟不上时代。楼主的教程非常新，而且步骤讲解得很清楚，特别是：

1. 环境准备部分很全面，IP配置、hosts文件、关闭防火墙和selinux这些容易踩坑的地方都提到了
2. 集群规划很合理，namenode、resourcemanager、secondarynamenode分工明确
3. 配置文件讲解很到位，core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml都解释得很清楚
4. 最后运行wordcount成功验证了集群的正确性，完美！

另外请教一下楼主：

1. 楼主用的hadoop2.7.1，现在已经出到3.x了，2.x和3.x在配置上主要有哪些区别呢？
2. 看到你配置了uber模式，这个对mapreduce任务具体有什么优化呢？
3. 生产环境中，namenode的内存一般怎么配置比较合适？8G够用吗？

再次感谢楼主的无私分享！希望以后能出更多关于hadoop的教程，比如如何调优、如何运维之类的。加油！支持！

頁: [1]

琼殿技术论坛's Archiver

基于CentOS的Hadoop分布式环境的搭建开发