CentOS安装Hadoop
<p>Hadoop的核心由3个部分组成：</p><p> HDFS: Hadoop Distributed File System，分布式文件系统，hdfs还可以再细分为NameNode、SecondaryNameNode、DataNode。</p>
<p> YARN: Yet Another Resource Negotiator，资源管理调度系统</p>
<p> Mapreduce：分布式运算框架</p>
<p> </p>
<p>1、软件与环境</p>
<p> 环境：CentOS-7-x86_64-Minimal-1810</p>
<p> hadoop版本：jdk-8u221-linux-x64.tar.gz，下载地址：https://www.apache.org/dist/hadoop/common/</p>
<p> jdk版本：jdk-8u221-linux-x64.tar.gz，hadoop只支持jdk7和jdk8，不支持jdk11</p>
<p>2、解压安装文件</p>
<p> 通过ftp等工具讲安装包上传到服务器上，并解压到/usr/local/目录</p>
<div class="cnblogs_code">
<pre>cd /usr/local/<span style="color: rgba(0, 0, 0, 1)">
tar </span>-zxvf /<span style="color: rgba(0, 0, 255, 1)">var</span>/ftp/pub/jdk-8u221-linux-<span style="color: rgba(0, 0, 0, 1)">x64.tar.gz
tar </span>-zxvf /<span style="color: rgba(0, 0, 255, 1)">var</span>/ftp/pub/hadoop-<span style="color: rgba(128, 0, 128, 1)">2.9</span>.<span style="color: rgba(128, 0, 128, 1)">2</span>.tar.gz</pre>
</div>
<p>3、配置JDK</p>
<p> 修改${HADOOP_HMOE}/etc/hadoop/hadoop-env.sh文件，修改JAVA_HOME配置（也可以修改/etc/profile文件，增加JAVA_HOME配置）。</p>
<div class="cnblogs_code">
<pre><span style="color: rgba(0, 0, 0, 1)">vi etc/hadoop/hadoop-env.sh
// 修改为
export JAVA_HOME=/usr/local/jdk1.8.0_221/</span></pre>
</div>
<p>4、设置伪分布模式（Pseudo-Distributed Operation）</p>
<p> 修改etc/hadoop/core-site.xml文件，增加配置（fs.defaultFS：默认文件系统名称）：</p>
<div class="cnblogs_code">
<pre><span style="color: rgba(0, 0, 255, 1)"><</span><span style="color: rgba(128, 0, 0, 1)">configuration</span><span style="color: rgba(0, 0, 255, 1)">></span>
<span style="color: rgba(0, 0, 255, 1)"><</span><span style="color: rgba(128, 0, 0, 1)">property</span><span style="color: rgba(0, 0, 255, 1)">></span>
<span style="color: rgba(0, 0, 255, 1)"><</span><span style="color: rgba(128, 0, 0, 1)">name</span><span style="color: rgba(0, 0, 255, 1)">></span>fs.defaultFS<span style="color: rgba(0, 0, 255, 1)"></</span><span style="color: rgba(128, 0, 0, 1)">name</span><span style="color: rgba(0, 0, 255, 1)">></span>
<span style="color: rgba(0, 0, 255, 1)"><</span><span style="color: rgba(128, 0, 0, 1)">value</span><span style="color: rgba(0, 0, 255, 1)">></span>hdfs://localhost:9000<span style="color: rgba(0, 0, 255, 1)"></</span><span style="color: rgba(128, 0, 0, 1)">value</span><span style="color: rgba(0, 0, 255, 1)">></span>
<span style="color: rgba(0, 0, 255, 1)"></</span><span style="color: rgba(128, 0, 0, 1)">property</span><span style="color: rgba(0, 0, 255, 1)">></span>
<span style="color: rgba(0, 0, 255, 1)"></</span><span style="color: rgba(128, 0, 0, 1)">configuration</span><span style="color: rgba(0, 0, 255, 1)">></span></pre>
</div>
<p> 修改etc/hadoop/hdfs-site.xml文件，增加配置（dfs.replication：文件副本数）：</p>
<div class="cnblogs_code">
<pre><span style="color: rgba(0, 0, 255, 1)"><</span><span style="color: rgba(128, 0, 0, 1)">configuration</span><span style="color: rgba(0, 0, 255, 1)">></span>
<span style="color: rgba(0, 0, 255, 1)"><</span><span style="color: rgba(128, 0, 0, 1)">property</span><span style="color: rgba(0, 0, 255, 1)">></span>
<span style="color: rgba(0, 0, 255, 1)"><</span><span style="color: rgba(128, 0, 0, 1)">name</span><span style="color: rgba(0, 0, 255, 1)">></span>dfs.replication<span style="color: rgba(0, 0, 255, 1)"></</span><span style="color: rgba(128, 0, 0, 1)">name</span><span style="color: rgba(0, 0, 255, 1)">></span>
<span style="color: rgba(0, 0, 255, 1)"><</span><span style="color: rgba(128, 0, 0, 1)">value</span><span style="color: rgba(0, 0, 255, 1)">></span>1<span style="color: rgba(0, 0, 255, 1)"></</span><span style="color: rgba(128, 0, 0, 1)">value</span><span style="color: rgba(0, 0, 255, 1)">></span>
<span style="color: rgba(0, 0, 255, 1)"></</span><span style="color: rgba(128, 0, 0, 1)">property</span><span style="color: rgba(0, 0, 255, 1)">></span>
<span style="color: rgba(0, 0, 255, 1)"></</span><span style="color: rgba(128, 0, 0, 1)">configuration</span><span style="color: rgba(0, 0, 255, 1)">></span></pre>
</div>
<p>5、设置主机允许无密码SSH链接</p>
<div class="cnblogs_code">
<pre><span style="color: rgba(0, 0, 0, 1)">ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa // 创建公钥私钥对
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys //
chmod 0600 ~/.ssh/authorized_keys // 设置权限，owner有读写权限，group和other无权限</span></pre>
</div>
<p>6、格式化文件系统</p>
<div class="cnblogs_code">
<pre>bin/hdfs namenode -format</pre>
</div>
<p>7、启动NameNode和DataNode进程（启动hdfs）</p>
<div class="cnblogs_code">
<pre><span style="color: rgba(0, 0, 0, 1)">./sbin/start-dfs.sh // 启动NameNode和DataNode进程
./sbin/stop-dfs.sh// 关闭NameNode和DataNode进程</span></pre>
</div>
<p><img src="https://img2018.cnblogs.com/blog/1031555/201909/1031555-20190910131638533-874768126.png"></p>
<p> 输入地址：http://192.168.114.135:50070，可查看HDFS</p>
<p><img src="https://img2018.cnblogs.com/blog/1031555/201909/1031555-20190910133312208-1282505704.png"></p>
<p>8、 启动YARN</p>
<div class="cnblogs_code">
<pre><span style="color: rgba(0, 0, 0, 1)">./sbin/start-yarn.sh
./sbin/stop-yarn.sh</span></pre>
</div>
<p><img src="https://img2018.cnblogs.com/blog/1031555/201909/1031555-20190910174331646-1743282090.png"></p>
<p> </p>
<p> 输入地址：http://192.168.114.135:8088/，可查看YARN</p>
<p><img src="https://img2018.cnblogs.com/blog/1031555/201909/1031555-20190910173940884-762261211.png"></p>
<p> </p><br><br>
来源：https://www.cnblogs.com/zhi-leaf/p/11496877.html
頁:
[1]