Hadoop 2.x伪分布式环境搭建详细步骤
<p>本文以图文结合的方式详细介绍了Hadoop 2.x伪分布式环境搭建的全过程，供大家参考，具体内容如下</p>
<p>
<strong>1、修改hadoop-env.sh、yarn-env.sh、mapred-env.sh</strong></p>
<p>
<span><strong>方法：</strong></span>使用notepad++(beifeng用户)打开这三个文件</p>
<p>
<span><strong>添加代码：</strong></span>export JAVA_HOME=/opt/modules/jdk1.7.0_67</p>
<p>
<strong>2、修改</strong>core-site.xml、hdfs-site.xml、yarn-site.xml、mapred-site.xml<strong>配置文件</strong></p>
<p>
1）修改core-site.xml</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterxml" id="highlighter_962673">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="xml plain"><</code><code class="xml keyword">configuration</code><code class="xml plain">></code>
</div>
<div class="line number2 index1 alt1">
<code class="xml plain"> <</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number3 index2 alt2">
<code class="xml plain"> <</code><code class="xml keyword">name</code><code class="xml plain">>fs.defaultFS</</code><code class="xml keyword">name</code><code class="xml plain">></code>
</div>
<div class="line number4 index3 alt1">
<code class="xml plain"> <</code><code class="xml keyword">value</code><code class="xml plain">>hdfs://Hadoop-senior02.beifeng.com:8020</</code><code class="xml keyword">value</code><code class="xml plain">></code>
</div>
<div class="line number5 index4 alt2">
<code class="xml plain"> </</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number6 index5 alt1">
<code class="xml plain"> <</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number7 index6 alt2">
<code class="xml plain"> <</code><code class="xml keyword">name</code><code class="xml plain">>hadoop.tmp.dir</</code><code class="xml keyword">name</code><code class="xml plain">></code>
</div>
<div class="line number8 index7 alt1">
<code class="xml plain"> <</code><code class="xml keyword">value</code><code class="xml plain">>/opt/modules/hadoop-2.5.0/data</</code><code class="xml keyword">value</code><code class="xml plain">></code>
</div>
<div class="line number9 index8 alt2">
<code class="xml plain"> </</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number10 index9 alt1">
<code class="xml plain"></</code><code class="xml keyword">configuration</code><code class="xml plain">></code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
2）修改hdfs-site.xml</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterxml" id="highlighter_337851">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="xml plain"><</code><code class="xml keyword">configuration</code><code class="xml plain">></code>
</div>
<div class="line number2 index1 alt1">
<code class="xml plain"> <</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number3 index2 alt2">
<code class="xml plain"> <</code><code class="xml keyword">name</code><code class="xml plain">>dfs.replication</</code><code class="xml keyword">name</code><code class="xml plain">></code>
</div>
<div class="line number4 index3 alt1">
<code class="xml plain"> <</code><code class="xml keyword">value</code><code class="xml plain">>1</</code><code class="xml keyword">value</code><code class="xml plain">></code>
</div>
<div class="line number5 index4 alt2">
<code class="xml plain"> </</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number6 index5 alt1">
<code class="xml plain"> <</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number7 index6 alt2">
<code class="xml plain"> <</code><code class="xml keyword">name</code><code class="xml plain">>dfs.namenode.http-address</</code><code class="xml keyword">name</code><code class="xml plain">></code>
</div>
<div class="line number8 index7 alt1">
<code class="xml plain"> <</code><code class="xml keyword">value</code><code class="xml plain">>Hadoop-senior02.beifeng.com:50070</</code><code class="xml keyword">value</code><code class="xml plain">></code>
</div>
<div class="line number9 index8 alt2">
<code class="xml plain"> </</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number10 index9 alt1">
<code class="xml plain"></</code><code class="xml keyword">configuration</code><code class="xml plain">></code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
3）修改yarn-site.xml</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterxml" id="highlighter_530058">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
<div class="line number15 index14 alt2">
15</div>
<div class="line number16 index15 alt1">
16</div>
<div class="line number17 index16 alt2">
17</div>
<div class="line number18 index17 alt1">
18</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="xml plain"><</code><code class="xml keyword">configuration</code><code class="xml plain">></code>
</div>
<div class="line number2 index1 alt1">
<code class="xml plain"> <</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number3 index2 alt2">
<code class="xml plain"> <</code><code class="xml keyword">name</code><code class="xml plain">>yarn.nodemanager.aux-services</</code><code class="xml keyword">name</code><code class="xml plain">></code>
</div>
<div class="line number4 index3 alt1">
<code class="xml plain"> <</code><code class="xml keyword">value</code><code class="xml plain">>mapreduce_shuffle</</code><code class="xml keyword">value</code><code class="xml plain">></code>
</div>
<div class="line number5 index4 alt2">
<code class="xml plain"> </</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number6 index5 alt1">
<code class="xml plain"> <</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number7 index6 alt2">
<code class="xml plain"> <</code><code class="xml keyword">name</code><code class="xml plain">>yarn.resourcemanager.hostname</</code><code class="xml keyword">name</code><code class="xml plain">></code>
</div>
<div class="line number8 index7 alt1">
<code class="xml plain"> <</code><code class="xml keyword">value</code><code class="xml plain">>Hadoop-senior02.beifeng.com</</code><code class="xml keyword">value</code><code class="xml plain">></code>
</div>
<div class="line number9 index8 alt2">
<code class="xml plain"> </</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number10 index9 alt1">
<code class="xml plain"> <</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number11 index10 alt2">
<code class="xml plain"> <</code><code class="xml keyword">name</code><code class="xml plain">>yarn.log-aggregation-enable</</code><code class="xml keyword">name</code><code class="xml plain">></code>
</div>
<div class="line number12 index11 alt1">
<code class="xml plain"> <</code><code class="xml keyword">value</code><code class="xml plain">>true</</code><code class="xml keyword">value</code><code class="xml plain">></code>
</div>
<div class="line number13 index12 alt2">
<code class="xml plain"> </</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number14 index13 alt1">
<code class="xml plain"> <</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number15 index14 alt2">
<code class="xml plain"> <</code><code class="xml keyword">name</code><code class="xml plain">>yarn.log-aggregation.retain-seconds</</code><code class="xml keyword">name</code><code class="xml plain">></code>
</div>
<div class="line number16 index15 alt1">
<code class="xml plain"> <</code><code class="xml keyword">value</code><code class="xml plain">>86400</</code><code class="xml keyword">value</code><code class="xml plain">></code>
</div>
<div class="line number17 index16 alt2">
<code class="xml plain"> </</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number18 index17 alt1">
<code class="xml plain"></</code><code class="xml keyword">configuration</code><code class="xml plain">></code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
4）修改mapred-site.xml</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterxml" id="highlighter_951671">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="xml plain"><</code><code class="xml keyword">configuration</code><code class="xml plain">></code>
</div>
<div class="line number2 index1 alt1">
<code class="xml plain"> <</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number3 index2 alt2">
<code class="xml plain"> <</code><code class="xml keyword">name</code><code class="xml plain">>mapreduce.framework.name</</code><code class="xml keyword">name</code><code class="xml plain">></code>
</div>
<div class="line number4 index3 alt1">
<code class="xml plain"> <</code><code class="xml keyword">value</code><code class="xml plain">>yarn</</code><code class="xml keyword">value</code><code class="xml plain">></code>
</div>
<div class="line number5 index4 alt2">
<code class="xml plain"> </</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number6 index5 alt1">
<code class="xml plain"> <</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number7 index6 alt2">
<code class="xml plain"> <</code><code class="xml keyword">name</code><code class="xml plain">>mapreduce.jobhistory.webapp.address</</code><code class="xml keyword">name</code><code class="xml plain">></code>
</div>
<div class="line number8 index7 alt1">
<code class="xml plain"> <</code><code class="xml keyword">value</code><code class="xml plain">>0.0.0.0:19888</</code><code class="xml keyword">value</code><code class="xml plain">></code>
</div>
<div class="line number9 index8 alt2">
<code class="xml plain"> </</code><code class="xml keyword">property</code><code class="xml plain">></code>
</div>
<div class="line number10 index9 alt1">
<code class="xml plain"></</code><code class="xml keyword">configuration</code><code class="xml plain">></code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
<strong>3、启动hdfs</strong></p>
<p>
1）格式化namenode:$ bin/hdfs namenode -format</p>
<p>
2）启动namenode:$sbin/hadoop-daemon.sh start namenode</p>
<p>
3）启动datanode:$sbin/hadoop-daemon.sh start datanode</p>
<p>
4）hdfs监控web页面：http://hadoop-senior02.beifeng.com:50070</p>
<p>
<strong>4、启动yarn</strong></p>
<p>
1）启动resourcemanager:$sbin/yarn-daemon.sh start resourcemanager</p>
<p>
2）启动nodemanager:sbin/yarn-daemon.sh start nodemanager</p>
<p>
3）yarn监控web页面：http://hadoop-senior02.beifeng.com:8088</p>
<p>
<strong>5、测试wordcount jar包</strong></p>
<p>
1）定位路径：/opt/modules/hadoop-2.5.0</p>
<p>
2）代码测试：bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar wordcount /input/sort.txt /output6/</p>
<p>
运行过程：</p>
<p>
<span>16/05/08 06:39:13 INFO client.RMProxy: Connecting to ResourceManager at Hadoop-senior02.beifeng.com/192.168.241.130:8032<br>
16/05/08 06:39:15 INFO input.FileInputFormat: Total input paths to process : 1<br>
16/05/08 06:39:15 INFO mapreduce.JobSubmitter: number of splits:1<br>
16/05/08 06:39:15 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1462660542807_0001<br>
16/05/08 06:39:16 INFO impl.YarnClientImpl: Submitted application application_1462660542807_0001<br>
16/05/08 06:39:16 INFO mapreduce.Job: The url to track the job: http://Hadoop-senior02.beifeng.com:8088/proxy/application_1462660542807_0001/<br>
16/05/08 06:39:16 INFO mapreduce.Job: Running job: job_1462660542807_0001<br>
16/05/08 06:39:36 INFO mapreduce.Job: Job job_1462660542807_0001 running in uber mode : false<br>
16/05/08 06:39:36 INFO mapreduce.Job: map 0% reduce 0%<br>
16/05/08 06:39:48 INFO mapreduce.Job: map 100% reduce 0%<br>
16/05/08 06:40:04 INFO mapreduce.Job: map 100% reduce 100%<br>
16/05/08 06:40:04 INFO mapreduce.Job: Job job_1462660542807_0001 completed successfully<br>
16/05/08 06:40:04 INFO mapreduce.Job: Counters: 49</span></p>
<p>
3）结果查看：bin/hdfs dfs -text /output6/par*</p>
<p>
运行结果：</p>
<p>
<span>hadoop 2<br>
jps 1<br>
mapreduce 2<br>
yarn 1</span></p>
<p>
<strong>6、MapReduce历史服务器</strong></p>
<p>
1）启动：sbin/mr-jobhistory-daemon.sh start historyserver</p>
<p>
2）web ui界面：http://hadoop-senior02.beifeng.com:19888</p>
<p>
<strong>7、hdfs、yarn、mapreduce功能</strong></p>
<p>
<strong><span>1）hdfs：</span></strong>分布式文件系统，高容错性的文件系统，适合部署在廉价的机器上。</p>
<p>
<strong>hdfs</strong>是一个主从结构，分为namenode和datanode，其中namenode是命名空间，datanode是存储空间，datanode以数据块的形式进行存储，每个数据块128M</p>
<p>
<span><strong>2）yarn：</strong></span>通用资源管理系统，为上层应用提供统一的资源管理和调度。</p>
<p>
<strong>yarn</strong>分为resourcemanager和nodemanager，resourcemanager负责资源调度和分配，nodemanager负责数据处理和资源</p>
<p>
<span><strong>3）mapreduce：</strong></span>MapReduce是一种计算模型，分为Map（映射）和Reduce（归约）。</p>
<p>
<strong>map</strong>将每一行数据处理后，以键值对的形式出现，并传给reduce；reduce将map传过来的数据进行汇总和统计。</p>
<p>
以上就是本文的全部内容，希望对大家的学习有所帮助。</p>
頁:
[1]