Linux中Centos7搭建Hadoop服务步骤

有明發表於 2023-10-10 00:00:00

<p>
下载Hadoop 官网：http://hadoop.apache.org/releases.html</p><p>
先配置jdk环境</p><p>
下载以后解压到到/usr/local</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_459169"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">tar -zxvf hadoop-2.8.0.tar.gz -c /usr/local</code></div></div></td></tr></tbody></table></div></div></div><p>
为了方便操作把hadoop-2.8.0 改为hadoop</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_716378"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">mv /usr/local/hadoop-2.8.0 /usr/local/hadoop</code></div></div></td></tr></tbody></table></div></div></div><p>
查看主机名</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_381190"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">hostname //第一个参数为主机名</code></div></div></td></tr></tbody></table></div></div></div><p>
检查是否可以免密码</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_663070"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">ssh localhost //这里的localhost为主机名</code></div></div></td></tr></tbody></table></div></div></div><p>
注意：一般初次安装都需要密码</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_771080"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">ssh-keygen -t dsa -p '' -f ~/.ssh/id_dsa</code></div></div></td></tr></tbody></table></div></div></div><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_595642"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys</code></div></div></td></tr></tbody></table></div></div></div><p>
再次验证</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_617696"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">ssh-keygen -t dsa -p '' -f ~/.ssh/id_dsa</code></div></div></td></tr></tbody></table></div></div></div><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_341639"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys</code></div></div></td></tr></tbody></table></div></div></div><p>
注意：如没有enter password 就说明免密码了</p><p>
配置环境变量</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_1829"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">vim /etc/profile</code></div></div></td></tr></tbody></table></div></div></div><p>
末尾添加</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_137475"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div><div class="line number2 index1 alt1">
2</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">export hadoop_home=/usr/local/hadoop </code></div><div class="line number2 index1 alt1"><code class="xhtml plain">export path=$java_home/bin:$path:$home/bin:$hadoop_home/bin</code></div></div></td></tr></tbody></table></div></div></div><p>
使环境变量立即生效</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_526174"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">source /etc/profile</code></div></div></td></tr></tbody></table></div></div></div><p>
创建hadoop的临时文件存放地</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_216457"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">mkdir /usr/local/hadoop/tmp</code></div></div></td></tr></tbody></table></div></div></div><p>
编辑hadoop的配置文件</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_498780"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div><div class="line number2 index1 alt1">
2</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">cd /usr/local/hadoop/etc/hadoop/</code></div><div class="line number2 index1 alt1"><code class="xhtml plain">vim hadoop-env.sh</code></div></div></td></tr></tbody></table></div></div></div><p>
末尾添加</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_211870"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div><div class="line number2 index1 alt1">
2</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">export java_home=/usr/local/java/jdk1.8.0_131/ </code></div><div class="line number2 index1 alt1"><code class="xhtml plain">export hadoop_conf_dir=/usr/local/hadoop/etc/hadoop</code></div></div></td></tr></tbody></table></div></div></div><p>
注意：路径要对</p><p>
使环境变量立即生效</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_871600"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">source hadoop-env.sh</code></div></div></td></tr></tbody></table></div></div></div><p>
配置另一个文件</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_901907"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">vim core-site.xml</code></div></div></td></tr></tbody></table></div></div></div><p>
在<configuration></configuration>中添加以下内容</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_663820"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div><div class="line number2 index1 alt1">
2</div><div class="line number3 index2 alt2">
3</div><div class="line number4 index3 alt1">
4</div><div class="line number5 index4 alt2">
5</div><div class="line number6 index5 alt1">
6</div><div class="line number7 index6 alt2">
7</div><div class="line number8 index7 alt1">
8</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain"><</code><code class="xhtml keyword">property</code><code class="xhtml plain">> </code></div><div class="line number2 index1 alt1"><code class="xhtml spaces">    </code><code class="xhtml plain"><</code><code class="xhtml keyword">name</code><code class="xhtml plain">>fs.defaultfs<code class="xhtml keyword">name</code><code class="xhtml plain">> </code></code></div><div class="line number3 index2 alt2"><code class="xhtml spaces">    </code><code class="xhtml plain"><</code><code class="xhtml keyword">value</code><code class="xhtml plain">>hdfs://localhost:9000<code class="xhtml keyword">value</code><code class="xhtml plain">> </code></code></div><div class="line number4 index3 alt1"><code class="xhtml spaces">  </code><code class="xhtml plain"><code class="xhtml keyword">property</code><code class="xhtml plain">> </code></code></div><div class="line number5 index4 alt2"><code class="xhtml plain"><</code><code class="xhtml keyword">property</code><code class="xhtml plain">> </code></div><div class="line number6 index5 alt1"><code class="xhtml plain"><</code><code class="xhtml keyword">name</code><code class="xhtml plain">>hadoop.tmp.dir<code class="xhtml keyword">name</code><code class="xhtml plain">> </code></code></div><div class="line number7 index6 alt2"><code class="xhtml plain"><</code><code class="xhtml keyword">value</code><code class="xhtml plain">>file:/usr/local/hadoop/tmp<code class="xhtml keyword">value</code><code class="xhtml plain">> </code></code></div><div class="line number8 index7 alt1"><code class="xhtml spaces">    </code><code class="xhtml plain"><code class="xhtml keyword">property</code><code class="xhtml plain">></code></code></div></div></td></tr></tbody></table></div></div></div><p>
配置下一个文件 hdfs-site.xml</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_136629"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">vim hdfs-site.xml</code></div></div></td></tr></tbody></table></div></div></div><p>
在<configuration></configuration>中添加以下内容</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_726097"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div><div class="line number2 index1 alt1">
2</div><div class="line number3 index2 alt2">
3</div><div class="line number4 index3 alt1">
4</div><div class="line number5 index4 alt2">
5</div><div class="line number6 index5 alt1">
6</div><div class="line number7 index6 alt2">
7</div><div class="line number8 index7 alt1">
8</div><div class="line number9 index8 alt2">
9</div><div class="line number10 index9 alt1">
10</div><div class="line number11 index10 alt2">
11</div><div class="line number12 index11 alt1">
12</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain"><</code><code class="xhtml keyword">property</code><code class="xhtml plain">> </code></div><div class="line number2 index1 alt1"><code class="xhtml spaces">    </code><code class="xhtml plain"><</code><code class="xhtml keyword">name</code><code class="xhtml plain">>dfs.replication<code class="xhtml keyword">name</code><code class="xhtml plain">> </code></code></div><div class="line number3 index2 alt2"><code class="xhtml spaces">    </code><code class="xhtml plain"><</code><code class="xhtml keyword">value</code><code class="xhtml plain">>1<code class="xhtml keyword">value</code><code class="xhtml plain">> </code></code></div><div class="line number4 index3 alt1"><code class="xhtml spaces">  </code><code class="xhtml plain"><code class="xhtml keyword">property</code><code class="xhtml plain">> </code></code></div><div class="line number5 index4 alt2"><code class="xhtml spaces"> </code><code class="xhtml plain"><</code><code class="xhtml keyword">property</code><code class="xhtml plain">> </code></div><div class="line number6 index5 alt1"><code class="xhtml spaces">    </code><code class="xhtml plain"><</code><code class="xhtml keyword">name</code><code class="xhtml plain">>dfs.namenode.name.dir<code class="xhtml keyword">name</code><code class="xhtml plain">> </code></code></div><div class="line number7 index6 alt2"><code class="xhtml spaces">    </code><code class="xhtml plain"><</code><code class="xhtml keyword">value</code><code class="xhtml plain">>file:/usr/local/hadoop/tmp/dfs/name<code class="xhtml keyword">value</code><code class="xhtml plain">> </code></code></div><div class="line number8 index7 alt1"><code class="xhtml spaces">    </code><code class="xhtml plain"><code class="xhtml keyword">property</code><code class="xhtml plain">> </code></code></div><div class="line number9 index8 alt2"><code class="xhtml spaces">    </code><code class="xhtml plain"><</code><code class="xhtml keyword">property</code><code class="xhtml plain">> </code></div><div class="line number10 index9 alt1"><code class="xhtml spaces">     </code><code class="xhtml plain"><</code><code class="xhtml keyword">name</code><code class="xhtml plain">>dfs.datanode.data.dir<code class="xhtml keyword">name</code><code class="xhtml plain">> </code></code></div><div class="line number11 index10 alt2"><code class="xhtml spaces">     </code><code class="xhtml plain"><</code><code class="xhtml keyword">value</code><code class="xhtml plain">>file:/usr/local/hadoop/tmp/dfs/data<code class="xhtml keyword">value</code><code class="xhtml plain">> </code></code></div><div class="line number12 index11 alt1"><code class="xhtml spaces">    </code><code class="xhtml plain"><code class="xhtml keyword">property</code><code class="xhtml plain">></code></code></div></div></td></tr></tbody></table></div></div></div><p>
配置下一个文件mapred-site.xml 因为这个文件默认不存在我们把mapred-site.xml.template作为模板来配置</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_139487"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">cp mapred-site.xml.template mapred-site.xml</code></div></div></td></tr></tbody></table></div></div></div><p>
vim mapred-site.xml</p><p>
在<configuration></configuration>中添加以下内容</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_212566"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div><div class="line number2 index1 alt1">
2</div><div class="line number3 index2 alt2">
3</div><div class="line number4 index3 alt1">
4</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain"><</code><code class="xhtml keyword">property</code><code class="xhtml plain">> </code></div><div class="line number2 index1 alt1"><code class="xhtml spaces">    </code><code class="xhtml plain"><</code><code class="xhtml keyword">name</code><code class="xhtml plain">>mapreduce.framework.name<code class="xhtml keyword">name</code><code class="xhtml plain">> </code></code></div><div class="line number3 index2 alt2"><code class="xhtml spaces">    </code><code class="xhtml plain"><</code><code class="xhtml keyword">value</code><code class="xhtml plain">>yarn<code class="xhtml keyword">value</code><code class="xhtml plain">> </code></code></div><div class="line number4 index3 alt1"><code class="xhtml spaces">  </code><code class="xhtml plain"><code class="xhtml keyword">property</code><code class="xhtml plain">></code></code></div></div></td></tr></tbody></table></div></div></div><p>
配置下下个文件yarn-site.xml</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_248722"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">vim yarn-site.xml</code></div></div></td></tr></tbody></table></div></div></div><p>
在<configuration></configuration>中添加以下内容</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_705127"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div><div class="line number2 index1 alt1">
2</div><div class="line number3 index2 alt2">
3</div><div class="line number4 index3 alt1">
4</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain"><</code><code class="xhtml keyword">property</code><code class="xhtml plain">> </code></div><div class="line number2 index1 alt1"><code class="xhtml spaces">    </code><code class="xhtml plain"><</code><code class="xhtml keyword">name</code><code class="xhtml plain">>yarn.nodemanager.aux-services<code class="xhtml keyword">name</code><code class="xhtml plain">> </code></code></div><div class="line number3 index2 alt2"><code class="xhtml spaces">    </code><code class="xhtml plain"><</code><code class="xhtml keyword">value</code><code class="xhtml plain">>mapreduce_shuffle<code class="xhtml keyword">value</code><code class="xhtml plain">> </code></code></div><div class="line number4 index3 alt1"><code class="xhtml spaces">  </code><code class="xhtml plain"><code class="xhtml keyword">property</code><code class="xhtml plain">></code></code></div></div></td></tr></tbody></table></div></div></div><p>
配置最后一个文件yarn-env.sh</p><p>
vim yarn-env.sh</p><p>
注意：在第23行左右 export java_home （删除# 去掉注释），并更改正确jdk的路径</p><p>
格式化namenode</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_797978"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div><div class="line number2 index1 alt1">
2</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">cd /usr/local/hadoop</code></div><div class="line number2 index1 alt1"><code class="xhtml plain">bin/hdfs namenode-format</code></div></div></td></tr></tbody></table></div></div></div><p>
注意：成功的话，会看到 “successfully formatted” 和 “exitting withstatus 0” 的提示，若为 “exitting with status 1” 则是出错了</p><p>
通过脚本启动hdfs</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_116952"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">sbin/start-dfs.sh</code></div></div></td></tr></tbody></table></div></div></div><p>
打开浏览器访问http://localhost:50070，验证是否hdfs配置成功</p><p><img style="max-width:100%!important;height:auto!important;" title="Linux中Centos7搭建Hadoop服务步骤" alt="Linux中Centos7搭建Hadoop服务步骤" src="https://zhuji.jb51.net/uploads/img/202305/b8eefdf549fe7a6daf33d1fe4133ad1c.jpg"/></p><p>
再来启动yarn</p><div class="jb51code"><div><div class="syntaxhighlighterxhtml" id="highlighter_514790"><div class="toolbar">?</div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr class="firstRow"><td class="gutter"><div class="line number1 index0 alt2">
1</div></td><td class="code"><div class="container"><div class="line number1 index0 alt2"><code class="xhtml plain">sbin/start-yarn.sh</code></div></div></td></tr></tbody></table></div></div></div><p>
打开浏览器访问http://localhost:8088，验证yarn是否配置成功</p><p><img style="max-width:100%!important;height:auto!important;" title="Linux中Centos7搭建Hadoop服务步骤" alt="Linux中Centos7搭建Hadoop服务步骤" src="https://zhuji.jb51.net/uploads/img/202305/0477c4233bebdae3ee19fcfd50909879.jpg"/></p><p>
注意：由于浏览器存在缓存问题打开地址可能会失败关闭浏览器重新打开一两次就好，如果还是失败检查配置文件是否有误</p><p>
查看已启动的hadoop进程</p><p>
jps</p><p>
注意：如果没有 namenode 或 datanode ，那就是配置不成功</p><p><br/></p>

頁: [1]

琼殿技术论坛's Archiver

Linux中Centos7搭建Hadoop服务步骤