详解搭建ubuntu版hadoop集群

厚德致祥 發表於 2023-6-21 00:00:00

用到的工具：VMware、hadoop-2.7.2.tar、jdk-8u65-linux-x64.tar、ubuntu-16.04-desktop-amd64.iso

1、在VMware上安装ubuntu-16.04-desktop-amd64.iso

单击“创建虚拟机”è选择“典型（推荐安装）”è单击“下一步”

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/87ae88253aecb1b4c37a0b591b695d0b.jpg">

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/639c849964c375839a91175547d17a09.jpg">

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/6252e6ac524c870dbd568644e9d75bcf.jpg">

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/da7ad6eef3ea60cc5c89c855704bcf6b.jpg">

è点击完成

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/4a9c8f3aefcdcc47dac1ae49ffad4cc2.jpg">

修改/etc/hostname

vim hostname

保存退出

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/578c82d095ad814607f08491c747dcda.jpg">

修改etc/hosts
<div>
<pre>
<code class="hljs accesslog">
127.0.0.1localhost
192.168.1.100s100
192.168.1.101s101
192.168.1.102s102
192.168.1.103s103
192.168.1.104s104
192.168.1.105s105</code></pre>
</div>

配置NAT网络

查看window10下的ip地址及网关

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/2a104b5dbdaaa29692d0fb37319762b5.jpg">

配置/etc/network/interfaces
<div>
<pre>
<code class="hljs yaml">
#interfaces(5) file used by ifup(8) and ifdown(8)
#The loopback network interface
auto lo
iface lo inet loopback

#iface eth0 inet static
iface eth0 inet static
address 192.168.1.105
netmask 255.255.255.0
gateway 192.168.1.2
dns-nameservers 192.168.1.2
auto eth0
</code></pre>
</div>

也可以通过图形化界面配置

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/e151816b83601ba2888132f8f5910a01.jpg">

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/dd75ec8054e7e9eeabb2871f4465a244.jpg">



配置好后执行ping www.baidu.com看网络是不是已经起作用

当网络通了之后，要想客户机宿主机之前进行Ping通，只需要做以下配置

修改宿主机c:\windows\system32\drivers\etc\hosts文件

文件内容
<div>
<pre>
<code class="hljs accesslog">
127.0.0.1 localhost
192.168.1.100 s100
192.168.1.101 s101
192.168.1.102 s102
192.168.1.103 s103
192.168.1.104 s104
192.168.1.105 s105</code></pre>
</div>

安装ubuntu 163 14.04 源
<div>
<pre>
<code class="hljs shell">
$>cd /etc/apt/

$>gedit sources.list
</code></pre>
</div>

切记在配置之前做好备份
<div>
<pre>
<code class="hljs awk">
deb http://mirrors.163.com/ubuntu/ trusty main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ trusty-security main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ trusty-updates main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ trusty-backports main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ trusty main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ trusty-security main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ trusty-updates main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ trusty-backports main restricted universe multiverse</code></pre>
</div>

更新
<div>
<pre>
<code class="hljs pgsql">
$>apt-get update</code></pre>
</div>

在家根目录下新建soft文件夹 mkdir soft

但是建立完成后，该文件属于root用户，修改权限 chown enmoedu:enmoedu soft/

安装共享文件夹

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/263587771675124bbe74138ca1727cb5.jpg">

将该文件放到桌面，右键，点击“Extract here”

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/b92c5b68799ecbfe7ec904fb19c0eb58.jpg">

切换到enmoedu用户的家目录，cd /Desktop/vmware-tools-distrib

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/512c1b5df8e44f17660e5ef3ad926af5.jpg">

执行./vmware-install.pl文件

Enter键执行

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/1a52a125b88041d2f45b4fb9a3108bc0.jpg">

安装完成

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/28bc47936d2c105e11b447792e820715.jpg">

拷贝hadoop-2.7.2.tar、jdk-8u65-linux-x64.tar到enmoedu家目录下的/Downloads
<div>
<pre>
<code class="hljs stylus">
$> sudo cp hadoop-2.7.2.tar.gz jdk-8u65-linux-x64.tar.gz ~/Downloads/</code></pre>
</div>

分别解压hadoop-2.7.2.tar、jdk-8u65-linux-x64.tar到当前目录
<div>
<pre>
<code class="hljs angelscript">
$> tar -zxvf hadoop-2.7.2.tar.gz

$>tar -zxvf jdk-8u65-linux-x64.tar.gz

$>cp -r hadoop-2.7.2 /soft

$>cp -r jdk1.8.0_65/ /soft
</code></pre>
</div>

建立链接文件
<div>
<pre>
<code class="hljs angelscript">
$>ln -s hadoop-2.7.2/ hadoop

$>ln -s jdk1.8.0_65/ jdk

$>ls -ll
</code></pre>
</div>

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/f675009cf5f9d00b596ebc3e313b04a9.jpg">

配置环境变量
<div>
<pre>
<code class="hljs awk">
$>vim /etc/environment</code></pre>
</div>
<div>
<pre>
<code class="hljs ini">
JAVA_HOME=/soft/jdk
HADOOP_HOME=/soft/hadoop
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/soft/jdk/bin:/soft/hadoop/bin:/soft/hadoop/sbin"</code></pre>
</div>

让环境变量生效
<div>
<pre>
<code class="hljs mel">
$>source environment</code></pre>
</div>

检验安装是否成功
<div>
<pre>
<code class="hljs applescript">
$>java –version</code></pre>
</div>

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/8f164152d5f8f3f88f6afd9500eaa210.jpg">
<div>
<pre>
<code class="hljs applescript">
$>hadoop version</code></pre>
</div>

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/040381da0bd9f8ad1271e4584005536a.jpg">

配置/soft/hadoop/etc/hadoop/ 下的配置文件


<div>
<pre>
<code class="hljs dts">
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://s100/</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/enmoedu/hadoop</value>
</property>
</configuration></code></pre>
</div>


<div>
<pre>
<code class="hljs xml">
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>s104:50090</value>
<description>
The secondary namenode http server address and port.
</description>
</property>
</configuration></code></pre>
</div>


<div>
<pre>
<code class="hljs xml">
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration></code></pre>
</div>


<div>
<pre>
<code class="hljs xml">
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>s100</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration></code></pre>
</div>

配置ssh无密码登录

安装ssh
<div>
<pre>
<code class="hljs routeros">
$>sudo apt-get install ssh</code></pre>
</div>

生成秘钥对

在enmoedu家目录下执行
<div>
<pre>
<code class="hljs elixir">
$>ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa</code></pre>
</div>

导入公钥数据到授权库中
<div>
<pre>
<code class="hljs arcade">
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys</code></pre>
</div>

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/fc38d1029adca6c44684bcae30eeffa9.jpg">

测试localhost成功后，将master节点上的供钥拷贝到授权库中

其中root一样执行即可
<div>
<pre>
<code class="hljs elixir">
$>ssh localhost</code></pre>
</div>

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/0d1c8d036fae38d95aadc942ace9944f.jpg">

从master节点上测试是否成功。

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/4a9ff9fd7ea744f07ef9cf4341c4d333.jpg">

修改slaves文件


<div>
<pre>
<code class="hljs nginx">
s101
s102
s103
s105</code></pre>
</div>

其余机器，通过克隆，修改hostname和网络配置即可

塔建完成后

格式化hdfs文件系统
<div>
<pre>
<code class="hljs mel">
$>hadoop namenode –format</code></pre>
</div>

启动所有进程
<div>
<pre>
<code class="hljs pgsql">
start-all.sh</code></pre>
</div>

最终结果：

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/b29435c3c188108a2dac48419666e319.jpg">

自定义脚本xsync(在集群中分发文件)



循环复制文件到所有节点的相同目录下。


<div>
<pre>
<code class="hljs bash">
#!/bin/bash
pcount=$#
if (( pcount<1 ));then
echo no args;
exit;
fi
p1=$1;
fname=`basename $p1`
#echo $fname=$fname;

pdir=`cd -P $(dirname $p1) ; pwd`
#echo pdir=$pdir

cuser=`whoami`
for (( host=101;host<106;host=host+1 )); do
echo ------------s$host----------------
rsync -rvl $pdir/$fname $cuser@s$host:$pdir
done
</code></pre>
</div>

测试

xsync hello.txt

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/64291f9e86c0bdd29f765212d5340768.jpg">

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/04d3f802e3f971ccae84b6cc77292c04.jpg">

自定义脚本xcall(在所有主机上执行相同的命令)


<div>
<pre>
<code class="hljs bash">
#!/bin/bash
pcount=$#
if (( pcount<1 ));then
echo no args;
exit;
fi
echo -----------localhost----------------
$@
for (( host=101;host<106;host=host+1 )); do
echo ------------s$host-------------
ssh s$host $@

done
</code></pre>
</div>

测试 xcall rm –rf hello.txt

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/6e070506653b6213ba0a5dade730091d.jpg">

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/4bcabe9243c849ec68a8a5b9948f7697.jpg">

集群搭建完成后，测试次运行以下命令
<div>
<pre>
<code class="hljs stylus">
touch a.txt
gedit a.txt
hadoop fs -mkdir -p /user/enmoedu/data
hadoop fs -put a.txt /user/enmoedu/data
hadoop fs -lsr /</code></pre>
</div>

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/aa5340d623fce0ab1d31f075ca4b6044.jpg">

也可以进入浏览器查看

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/25d22de5c250351f0636ef6af40c4db9.jpg">

<img style="max-width:100%!important;height:auto!important;"title="详解搭建ubuntu版hadoop集群" alt="详解搭建ubuntu版hadoop集群" src="https://zhuji.jb51.net/uploads/img/202305/dcdc4b3678d7760fb05b53e419fc35ee.jpg">

以上就是本文的全部内容，希望对大家的学习有所帮助，也希望大家多多支持。

原文链接：http://www.cnblogs.com/cpyj/p/6664113.html?utm_source=tuicool&utm_medium=referral

頁: [1]

琼殿技术论坛's Archiver

详解搭建ubuntu版hadoop集群