rdma centos 7.3安装
<h1 class="artical-title">rdma centos 7.3安装</h1><div class="artical-title-list">
<div class="is-vip-bg-6 fl"><img class="is-vip-img is-vip-img-4 lazyload" alt="" data-uid="5908329" data-src="https://s3.51cto.com/wyfs02/M02/71/75/wKiom1XRMRSTTY41AABY7IDhS8U545_middle.jpg"></div>
corasql<span class="comment_number">0</span>人评论<span class="fr"><span class="fr">7680人阅读2017-05-28 16:29:40</span></span>
<div class="clear"> </div>
</div>
<div class="artical-content-bak main-content editor-side-new">
<div id="result" class="con editor-preview-side">
<h2>1、安装依赖包</h2>
<pre>yum install epel-release -y
yum install gcc gcc-c++ bc openssl-devel automake ncurses-devel libibverbs -y
yum install libibverbs-devel libibverbs-utils librdmacm librdmacm-devel librdmacm-utils perl-Switch elfutils-libelf-devel -y</pre>
<h2>2、 librxe-dev 和 rxe-dev下载</h2>
<p>下载地址</p>
<pre>Github: https://github.com/SoftRoCE/rxe-dev.git
Github: https://github.com/SoftRoCE/librxe-dev.git</pre>
<p>备注：rxe-dev下载v18版本，即rxe-dev-rxe<em>submission</em>v18</p>
<h2>3、安装rxe-dev</h2>
<pre>unzip rxe-dev-rxe_submission_v18.zip
cd rxe-dev-rxe_submission_v18/
cp /boot/config-3.10.0-514.el7.x86_64 .config</pre>
<p>备注：使用root用户，执行以下命令</p>
<pre>make menuconfig</pre>
<p>会出现选择界面（如果没出现，需要安装 ncurse-devel）<br>输入 "/" ，然后输入 rxe，按下 enter，会查找有关 rxe 的选择项。<br>输入数字 1，就会选择到“Software RDMA over Ethernet (ROCE) driver”的设置，输入 "M" ，选中 RDMA 的配置，如果 输不了 M，那就输入空格。<br>移动到保存按钮，回车，装保存到.config中，退出安装界面（exit）。<br>然后 vi .config 来确认 <br>CONFIG<em>RDMA</em>RXE 为 m<br>CONFIG<em>INFINIBAND<em>ADDR</em>TRANS 和 CONFIG</em>INFINIBAND<em>ADDR</em>TRANS_CONFIGFS 为 y</p>
<pre>make -j 4
make modules_install ，可能执行中途 会提示 丢失一些 module，这个 没关系，无关紧要。
make install
make headers_install INSTALL_HDR_PATH=/usr</pre>
<p>确认 新的内核是否在 grub 引导中。查看 /etc/grub.cfg 即可看见。在开机的时候可以选择 新内核启动</p>
<h2>4、安装 librxe-dev</h2>
<pre>cd librxe-dev
./configure --libdir=/usr/lib64/ --prefix=
make
make install<br><br><span style="font-size: 15px">Here is the issue:</span></pre>
<p><span style="font-size: 15px">checking for ibv_get_device_list in -libverbs... </span><br><span style="font-size: 15px">yes</span><br><span style="font-size: 15px">checking infiniband/driver.h usability... no</span><br><span style="font-size: 15px">checking infiniband/driver.h presence... no</span><br><span style="font-size: 15px">checking for infiniband/driver.h... no</span><br><span style="font-size: 15px">configure: error: <infiniband/driver.h> not found.librxe requires libibverbs.</span></p>
<p> </p>
<p><span style="font-size: 16px">How to fix?</span></p>
<div id="divCommentShow">
<div class="comment_my_posted">
<blockquote class="bq_post_comment"><span style="font-size: 16px">https://wangmingjun.com/2018/09/03/how-to-build-the-development-environment-of-software-rdma-over-converged-ethernet-roce/</span><br><br><span style="font-size: 16px">rxe-dev and librxe_dev these two repositories lack maintenance. And the RDMA-core has already contrained all the RXE utilities. So please use RDMA-core instead of librxe-dev.</span></blockquote>
<p> </p>
</div>
</div>
<p>重启操作系统，在开机启动时，选择4.7.0-rc3内核<br>启动后，查看内核版本</p>
<pre>uname -r</pre>
<h2>5、验证 rdma</h2>
<pre># rxe_cfg start
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
ens33 yes e1000
virbr0 no bridge
virbr0-nic no tun
# rxe_cfg add ens33
# rxe_cfg status
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
ens33 yes e1000 rxe0 1024 (3)
virbr0 no bridge
virbr0-nic no tun</pre>
<p>查看rxe设备<br>ibv<em>devices 程序显示该系统中目前所有设备，而 ibv</em>devinfo 命令会给出每个设备的具体信息</p>
<pre># ibv_devices
device node GUID
------ ----------------
rxe0 020c29fffe55c818
# ibv_devinfo rxe0
hca_id: rxe0
transport: InfiniBand (0)
fw_ver: 0.0.0
node_guid: 020c:29ff:fe55:c818
sys_p_w_picpath_guid: 0000:0000:0000:0000
vendor_id: 0x0000
vendor_part_id: 0
hw_ver: 0x0
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet</pre>
<h2>6、softRoCE连通性测试</h2>
<p>服务端</p>
<pre>rping -s -a 192.168.1.133 -v -C 10</pre>
<p>客户端</p>
<pre>rping -c -a 192.168.1.133 -v -C 10<br><br></pre>
<p>Test connectivity.</p>
<ul>
<li>On the server:</li>
</ul>
<table>
<tbody>
<tr>
<td class="gutter">
<pre><span class="line">1<br></span></pre>
</td>
<td class="code">
<pre><span class="line">ibv_rc_pingpong -d rxe0 -g 0<br></span></pre>
</td>
</tr>
</tbody>
</table>
<ul>
<li>On the client:</li>
</ul>
<table>
<tbody>
<tr>
<td class="gutter">
<pre><span class="line">1<br></span></pre>
</td>
<td class="code">
<pre><span class="line">ibv_rc_pingpong -d rxe0 -g 0 <server_management_ip><br></span></pre>
</td>
</tr>
</tbody>
</table>
<p>e.g Client:</p>
<table>
<tbody>
<tr>
<td class="gutter">
<pre><span class="line">1<br><span class="line">2<br><span class="line">3<br></span></span></span></pre>
</td>
<td class="code">
<pre><span class="line">kevin@ubuntu:~$ ibv_rc_pingpong -g 0 -d rxe0 -i 1 192.168.188.129<br><span class="line">local address:LID 0x0000, QPN 0x000011, PSN 0x2cd726, GID fe80::20c:29ff:febd:5e22<br><span class="line">remote address: LID 0x0000, QPN 0x000011, PSN 0x767a62, GID fe80::20c:29ff:fe44:4345</span></span></span></pre>
</td>
</tr>
</tbody>
</table>
<p>测试时需切换到root</p>
<p>Sever:</p>
<table>
<tbody>
<tr>
<td class="gutter">
<pre><span class="line">1<br></span></pre>
</td>
<td class="code">
<pre><span class="line">ib_send_bw –a<br></span></pre>
</td>
</tr>
</tbody>
</table>
<p>Client:</p>
<table>
<tbody>
<tr>
<td class="gutter">
<pre><span class="line">1<br></span></pre>
</td>
<td class="code">
<pre><span class="line">ib_send_bw 192.168.46.132 –a</span></pre>
</td>
</tr>
</tbody>
</table>
<h2>7、关于librdmacm编译说明</h2>
<pre>git clone https://github.com/ofiwg/librdmacm.git
cd librdmacm
yum install autoconf automake gettext gettext-devel libtool -y
./autogen.sh
./configure
make
make install</pre>
<h2>8、常见问题</h2>
<p>(1)如果你克隆虚机，需要解决网卡问题<br>(2)使用rdma，请将防火墙与selinx关闭</p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<h1 class="title">How to build the development environment of Software RDMA over Converged Ethernet (RoCE) ?</h1>
<h1>Soft-RoCE (RXE)</h1>
<p>In order to study the coding of RDMA, I need build the corresponding environment. Due to the lack of hardware, I found Soft-RoCE would be the first choice after some research work.</p>
<p>All the information focus on these two repositories: and . And others aim to help us how to build this virtual RDMA device which named RXE and how to use it.</p>
<h1>Struggle Against RXE</h1>
<p>Internet shows that the RXE need kernel support and user space codes.</p>
<p>Firstly, we git clone the to compile and install the new kernel to support RoCE. Then restart with the new kernel. Finally, compile the to get the utilities of Soft-Roce.</p>
<p>When I switched to the compiled new kernel, but it failed to start the system. When I configured the , it showed “configure: error: <infiniband/driver.h> not found. librxe requires libibverbs”.</p>
<p>This page also shows the same issue, and several people also hangs on this error.</p>
<h1>Sudden Inspiration</h1>
<p>MosesAlexander’s comment of “I just found that the rxe functionality is all in rdma-core now.” gives me sudden inspiration. The rdma-core has already contained all the RXE utilities, and it looks like that there’s no maintenance for the two repositories and .</p>
<h1>Solution</h1>
<p>Just “yum -y install libibverbs libibverbs-devel libibverbs-utils librdmacm librdmacm-devel librdmacm-utils” will be OK !</p>
<p><strong>Notice</strong>: I blog this post at 2018/09/03, based on CentOS 7 (3.10.0-862.el7.x86_64), whose kernel support RDMA-related technologies originally.</p>
<h1>Verification</h1>
<p>Run some commands to verify:</p>
<pre class="lang:default decode:true "># rxe_cfg start
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
ens33 yes e1000
virbr0 no bridge
virbr0-nic no tun
# rxe_cfg add ens33
# rxe_cfg status
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
ens33 yes e1000 rxe0 1024 (3)
virbr0 no bridge
virbr0-nic no tun
# ibv_devices
device node GUID
------ ----------------
rxe0 020c29fffe495c4d</pre>
<p>Also, you could run example codes from the-geek-in-the-corner.</p>
<p> </p>
<p>参考：</p>
<p>http://blog.sina.com.cn/s/blog_6de3aa8a0102wr14.html</p>
<p>http://www.unjeep.com/article/23742.html （rping测试，softroce/rdma安装测试）</p>
<p>https://github.com/SoftRoCE/rxe-dev/wiki/Validate-that-RXE-is-working （验证rxe是否工作）</p>
</div>
</div><br><br>
来源：https://www.cnblogs.com/zafu/p/11200148.html
頁:
[1]